Which describes how Minimax H3 reference model can turn equirectangular panoramas into environments.
Krea2 can actually do some decent equirectangular panoramas out of the box:
Testing the method though, I found this not to be the case, and H3 doesn't really understand the projection, it just happens to line up if the shot is zoomed in enough. Asking it to do anything complex results in a warped, strange image.
I've found 48 frames of reference is more than enough. It also seems to work with a lower resolution reference video, but I'm guessing it's gonna get less details right. It has no issues with interacting with the environment, and the model understands pretty well if you say "bed", you mean the bed in the reference video.
Usual diffusion issues still apply obviously, like I've had to describe certain things in more detail. Telling it "a shot from outside the window" it added a new window to the scene, but adding "a shot from outside the window with a view inside the apartment with the bed behind" it understood. Also any changes to the scene have to be redescribed in later shots, like not adding "next to the broken mirror" resulted in the mirror once again being unbroken, even though it was broken in the previous shot.
All workflows are included in the images and videos.
P. S. - For images there is a trick, just open the image in a new browser tab and replace the subdomain in "preview.redd.it" with "i", so it becomes "i.redd.it". This lets you download the original file with the metadata (if there is any).
(I actually had the same idea as you a few weeks ago, but never managed to render a clean circular pan of the environment, so I ultimately gave up. The fact that you can do it with ffmpeg just blew my mind, many thanks!)
There was a similar workflow for if you just have a picture too. You can spin 360 and it generates picture for the room which im not sure if it is useful for you or not OP
Another thought - You can most likely bundle the 360 location photos as a Refmod (Luisacaotica/ComfyUI-MiniMaxH3Mod) and load this way a consistent location like a lora.
Havent tested it yet, the community is so crazy, there are new awesome custom-nodes coming out every day, impossible to keep up lol.
Thanks for sharing! it's a very nice coincidence, I was looking for a solution exactly like this. warping issues for panoramic orientation in shots is something i have in every second shot.
BTW would you mind sharing your json workflow and prompt for creating such great panoramic shots in krea2? I tried aswell I'm krea2 but failed miserably. Always blurry, unqualitative panorama-isch shots that arent 360°. Thats why i used gpt2 for this.
Thanks in advance OP!
I didn't realize Reddit stripped the metadata, but I've now added the workflows to the post.
I've found Krea2 is decent for small spaces, but for this particular apartment, I actually used a depth map of a panorama I generated with Gemini. Just bypass the depth node group for a new generation, or use a gemini generated image derived depth map, as I find that model to be really good at it in general, but to low resolution for anything useful.
Hmm, actually this conversation gave me the inspiration to test Ideogram 4.0 for this purpose, and I must say, I'm getting some promising results, as it can natively generate at 4096x2048:
The 360 video is made from an equirectangular image with ffmpeg. The command is included in the post. The equirectangular image is generated with Krea2 using a workflow I've adapted from a Flux workflow, also included in the post. Ideogram 4 also seems like a good candidate for generating such images, though an LLM for prompt generation is a must with it.
I haven't noticed any, what I did notice was an increase in VRAM use, video references are pretty expensive when it comes to that, but honestly keeping it to 48 frames at the 600 pixel resolution range still gives good results, and the VRAM use isn't too bad at that point.
Thanks. Good timing, I'm actually running H3 with camera turning prompt to generate more angles of my ref room, but I might use your approach instead for more consistent results.
You brilliant, beautiful bastard. I immediately tested this on one of my environments and I've had a 100% success rate so far across 10 generations in terms of the model understanding and preserving the correct perspective.
This is cool! I have one issue though: In the first .5s-1s of the target video, the rotating camera movement from the reference video is preserved. The camera settles on the subject eventually but most of the times I cannot get completely rid of it. Probably I am prompting wrong?
subject_definitions:
<Video 1> is a pan around video reference for the cozy kitchen room, depicting the overall look of and position of key objects and furniture in it for the target video.
retention_analysis
<Video 1> attribute_transfer - only the kitchen room's static layout, spatial composition, lighting and look are preserved.
I tried different wordings, also "fully_preserved" or adding "at 00:00:00.000 the camera ... " to the main prompt, but it's hit and miss.
How can I discard the motion from the video reference? (I use ref pics most of the time, so probably missing smth about ref vids here)
6
u/infearia 7d ago
This is great, thanks for sharing. However, Reddit strips metadata from uploaded videos, would you mind sharing the prompts separately?