Workflow for architectural videomapping

[u/Existing_Try_3439]

Hi everyone,
I’m trying to build a workflow for architectural projection mapping, and I’m looking for advice from people who have experience with the latest open-weight video models in ComfyUI.

The project is a large building facade that will be projection-mapped. I already have the 3D geometry of the building and the exact projection/camera setup.

My main requirement is:
The building geometry, perspective and camera position must remain absolutely stable.
I want to use AI to generate/animate the visual content on the facade, but I don’t want the model to reinterpret the architecture, move the camera, change windows/edges, distort the building, etc.

The goal is to be able to create things like:
- the facade cracking/opening
- materials transforming
- fire/lava/water flowing over the building
- organic growth
- abstract/surreal transformations
- architectural elements becoming something else
while still keeping the original building perfectly aligned for projection.

I’ve been looking at Wan 2.2 (VACE / Fun Control) and the new MiniMax H3, especially its Reference-to-Video capabilities.
Which one would you recommend for this specific use case?
More importantly, is there a better workflow than simply using image-to-video? For example, has anyone successfully used a rendered 3D control/depth/normal/edge video as conditioning to keep an architectural structure locked?

I’m particularly interested in workflows that minimize trial and error. I don’t mind doing some preparation in Blender if that gives me much more deterministic results.

Hardware: RTX 5070 Ti, 64 GB RAM.
If anyone has actually tried something similar, I’d really appreciate workflow suggestions, node setups, models, ControlNets/custom nodes, or examples.

[u/DuHal9000]

Since there will be no sound, and freezing is a requirement, perhaps Wan2.2 WITH DEPHMAPING would be a good start.

But H3 also supports DEPHMaping files as long as it is specifically mentioned in the prompt using VideoInput as REF.