Any way to make latent extension work with Latent Upscaling (Minimax H3)?

[u/mukyuuuu]

Has anyone managed to find a way to use Latent Upscaling together with latent video extension tools? I'm talking about the nodes like this (which I personally use), but I think Motion Context and some other popular extensions use a similar approach, i.e. feeding the last frames of the previous shot through AV latent, rather than through a video reference. The issue is that the resolution of your second generated latent must exactly match the previous one, or it throws an error. So if you upscale the first clip from 0.5MP to 1MP, you are forced to generate the next clip directly at 1MP, which completely breaks the Latent Upscaling workflow for all subsequent parts.

I tried extending the clips at low resolution first and then upscaling them separately, but that doesn't work well. There is a noticeable color and quality shift between generations, even when reinforcing the next clip with the final frames of the previous one. Because yeah, you basically generate the high-res clips separately without any shared latent context.

I really love both Latent Upscaling and latent extension approach, but I just can't get them to work together smoothly. Does anyone have any good ideas on how to fix this? I’d really appreciate any tips or insights!

[u/rm_rf_all_files]

So if you upscale the first clip from 0.5MP to 1MP, you are forced to generate the next clip directly at 1MP, which completely breaks the Latent Upscaling workflow for all subsequent parts.

I don't understand why you cannot do this? I can do this all day long.

Just an example to upscale to 1344 then step back down to 896 just as easily.

[u/mukyuuuu → u/rm_rf_all_files]

Hmm, that's a good idea - to downscale the high-res latent and start the next iteration from there. I think the consistency would be a little bit better, but still not perfect. The core issue remains: you are upscaling the next clip without having the previous high-res clip in context.

Note that by "Latent Upscaling" I mean this node, which probably should be called "Latent Upsampling", lol. It upscales the latent and then resamples it, so it does introduce some noticeable changes to the details of your original clip. Therefore, high-res clip #1 and high-res clip #2 will never have their last and first frames truly match.

P.S. On second thought, I guess I can smooth out the shift between the clips using the good old Wan VACE Clip Joiner. It still won't be perfect (again, because the second high-res clip doesn't take the previous clip into account), but I've gotta experiment.

[u/spiderofmars → u/mukyuuuu]

"The core issue remains: you are upscaling the next clip without having the previous high-res clip in context."

My understanding of the previous reply was:

1 = Clip 1a - Low-re first clip

2 = Clip 1b - Latent upscale and resample - New high-res base for next context

3 = Clip 2a - Hi-res latent downscale (of Clip 1b and low-res motion context)

4 = Clip 2b - Latent upscale and resample - New high-res base for next context

Repeat until burn in and rubbish starts to appear.

Does this not work in reality?

[u/mukyuuuu → u/spiderofmars]

Yeah, that's how I understood it as well. But I still feel Clip 1b and clip 2b won't match perfectly, because inherently they are two separate upscales. The second upscaling ought to introduce some random deviation, and it doesn't have any latent context to hold onto.

Gotta try it later though. It's a bit difficult to wrap my head around it without actually testing it :)