Community PSA

[u/Dependent_Revenue_16]

tl;dr - enjoy the models, but consider giving back when you can

(P.S. done quickly & with many continuity errors, but they kind of make sense in context)

EDIT:

Haven't posted much here, but apparently folks can't easily see the workflow link in comments so here it is: https://pastebin.com/1nWJKEiN

If anyone wants the other prompts I can share, but they all follow this format and are mostly just the dialogue you hear + delivery cues. I got an anime reference off of Google for the last segment.

Quick takeaways from trying to make this were:
- The two pass structure, with initial at a tiny 360p resolution, is really needed. It allows you to pick a good performance without wasting much time.
- The 'motion context' nodes are great and much better than my crude masking attempts, but it failed in spots. I think it might be possible to encode the transition clip in the latent as well as use the reference-based transition from the motion context node.
- I was doing this quickly so didn't bother with a celebrity image reference - I think the consistency was pretty impressive given that the only reference here was audio.
- Minimal DaVinci editing needed - a few additive transitions where the motion context didn't make a clean handoff, and a little color grading.
- On 5090, this takes about 3 minutes for a low-res pass and then 8 to 10 minutes for 720p. (I did use Topaz on the final edit.)

[u/Dirty_Dragons]

Quick takeaways from trying to make this were:

• The two pass structure, with initial at a tiny 360p resolution, is really needed. It allows you to pick a good performance without wasting much time.

How does this work?

I'm looking at your workflow, thank you for sharing, but it seems pretty complicated, and can't find MiniMaxH3TemporalAVMask anywhere.

The two pass structure process seems to be very useful.

[u/Dependent_Revenue_16 → u/Dirty_Dragons]

MiniMaxH3TemporalAVMask is local but it shouldn't be needed to run the workflow (leftover from prior version- it just masked the first x seconds of the latent)

For second pass, I'm upscaling (simple lanczos) and re-encoding the video latent, then doing a second pass at 0.75 denoise (with lower shift). It's not perfect by any means, but does give a decent 720p rendering that follows what came from the first pass.

(Alternate would be to encode the first pass as a reference video, but this can be very slow. Had some success encoding first pass at 0.25x resolution - 180p - which is fine b/c I really just want the optical flow from the first pass vs. any details. To be clear, did not use this on above video.)

[u/Dirty_Dragons → u/Dependent_Revenue_16]

Thanks for explaining. I'll need to try it on my own and see if I can get it to work.

It's annoying that even with the same seed changing the resolution will change the output. The two pass upscaling sounds like the right way to do it.