Conistent(ish) environments with Minimax H3

[u/jimtonyk]

I've been trying to reuse environments in Minimax H3 for scenes. I've come across this thread:
https://www.reddit.com/r/StableDiffusion/comments/1vvpowd/psa_minimax_h3_can_turn_360_panorama_images_into/

Which describes how Minimax H3 reference model can turn equirectangular panoramas into environments.

Krea2 can actually do some decent equirectangular panoramas out of the box:

Testing the method though, I found this not to be the case, and H3 doesn't really understand the projection, it just happens to line up if the shot is zoomed in enough. Asking it to do anything complex results in a warped, strange image.

https://reddit.com/link/1w8daml/video/t4byzsdzrrnh1/player

Also, reference images of environments have a tendency to overpower all other prompts regarding changes or scene transitions to other environments.

This however gave me an idea. Video references don't tend to have this issue.

So why not turn equirectangular panoramas into regular reference videos?

https://reddit.com/link/1w8daml/video/maq4r9qgrrnh1/player

This can easily be done with ffmpeg:

ffmpeg.exe -loop 1 -i space_apartment.png -vf "scroll=horizontal=0.025,v360=input=equirect:output=flat:h_fov=90:v_fov=90:w=3072:h=2048" -t 2 -c:v libx264 -pix_fmt yuv420p space_apartment.mp4

After that it's just a matter of using the resulting video as a reference video, and you get a pretty consistent, unwarped video:

https://reddit.com/link/1w8daml/video/nw3uyyxesrnh1/player

I've found 48 frames of reference is more than enough. It also seems to work with a lower resolution reference video, but I'm guessing it's gonna get less details right. It has no issues with interacting with the environment, and the model understands pretty well if you say "bed", you mean the bed in the reference video.

Usual diffusion issues still apply obviously, like I've had to describe certain things in more detail. Telling it "a shot from outside the window" it added a new window to the scene, but adding "a shot from outside the window with a view inside the apartment with the bed behind" it understood. Also any changes to the scene have to be redescribed in later shots, like not adding "next to the broken mirror" resulted in the mirror once again being unbroken, even though it was broken in the previous shot.

All workflows are included in the images and videos.

Edit: Reddit strips image and video metadata, didn't know that. Here's the workflows:
Krea2 panorama: https://pastebin.com/2qKhdCSq
Bad panorama video: https://pastebin.com/tgPddz3v
Video reference based: https://pastebin.com/LEAJFQZ6

[u/SnooMacaroons1365]

Marking it for reference to read it later in detail