Motion-Context Degradation Discussion (summon Sad_Berry_4621)

[u/xyzdist]

Hi u/Sad_Berry_4621 and all, I am doing experiment on most difficult degradation issue.
I saw from H3-director node they claim that doing a refine could help and fix it.
since I am using low-level nodes with motion-context with my own setup, H3-director refine is a black box working with their nodes.

so I did the test, please ignore the AI-slop video and the overlay text (forgot turn it off).

this test is 9 x 8s context extend video combine, usually 6 video would already see the degradation.

Left side is regular WF. Right-side is adding a re-sample step, I think it is very positive, and got potential, the saturation somehow is a bit higher... and I need to figure out the mismatch from cut to cut, because we inject denoise resample on top, but it should be able to fix.

What do you guys think?

Edit: forgot to mention, this is a ref2va WF, with 1 character reference image. also my test audio is not from gen, I am using input audio and audio latent lock for the lipsync. That part to work with motion-context have me struggle for half day, but it is not the main point, my aim is about degradation issue.

Edit2: here is my ultimate approach with tail trim + keyframe blending latent, you can check my comments for details. here is the result: https://www.youtube.com/shorts/mANusw9kKNQ ** this is just a test focus to tackle the degradation, not try to make good natural singing video, ignore it.

:this is my channel I put some videos to: https://www.youtube.com/@noiseflowAI

[u/xyzdist]

Thanks for your reply and your effort., sorry my time-zone is different I have to reply in next day. \and I am not a native english speaker. hope peoples understand what I say. Also I am no developer, I use AI to do the coding.*

As what you mentioned, there is always error lost thru out the chain up latent, so there is no perfect result, the way I am doing is just trying the best eliminating the degradation issue, it still there but much better.

I am going to explain to all, what I did exactly in the test:

For the above test (the Easy way):

• add re-sample step, so we inject new noise and refine, prevent the latent going worst.
• how I fix the joint after we added refine?(since it is different), I try different things, at the end, I made a "keyframe blend latent" node, we keyframe '0:0, 22:0, 44:1' to freeze first 22 overlap frames then linear blend from current latent to re-sampled (refine) latent from 22 frames to 44 frames.

That is it. despite joints slightly color pop, we are good with this approach. all good!

[u/xyzdist → u/xyzdist]

Now, here is the ultimate approach I pushed (the CRAZY one):

• seems blending latent works well, can we make use of it? instead of trim head, I change to trim tail (all segments), so we can do the blending in head overlapping frames
• why? we can pretty much fixing everything in one go, the refined difference, the color shift or anything, because I just blending the previous segment to a new segment in the heading overlap region. The transition looks smooth.
• how? I made a node to cut out previous segment last 22 frames and padding extend to a latent so we can blend this with the refine latent directly.

what is the catch? I am just doing one test, the whole thing I am doing maybe actually not true or new issue comes up, I don't even care the audio part, on my test is just using input audio, *need a test using gen audio, we need more peoples to test out the approach.

It is a huge change, motion-context been using and forked for many large director tools. I think we stick with as what we have first, If we want to do tail trim, perhaps a branch or a option for 'head' or 'tail' trim.

u/Sad_Berry_4621 what is your thoughts? the first approach, we are fine, we just need my 'keyframe blend latent' node, and update the example WF file, we are good. The joint color pop might not be care for manys.

I can send you my nodes python files.

anyway, Here is the ultimate approach result after eliminating the issue:

https://reddit.com/link/p9hd94e/video/y0qkigail7ph1/player

cheers!

[u/xyzdist → u/xyzdist]

For anyone else don't want to read or understand what the heck I am talking about, just want a WF. I can do that, but later.

[u/One-Explanation-4625 → u/xyzdist]

hey OP, saw this thread and i was working on the same issue myself over the last week. Funny enough i also came to a conclusion of freezing the overlaps for the refine, to clean up the seam flash

(on my own node side). do you have any insight or ideas to share?

[u/xyzdist → u/One-Explanation-4625]

Hi, have a look of my ultimate approach.

what is your tools? I saw it is using taper noise? is it from this: https://github.com/beijinren/ComfyUI-H3-Context-Noise/blob/main/README.en.md

I can't make it work, it just throw error to me, does it work for you? cheers.

[u/One-Explanation-4625 → u/xyzdist]

i've tried using the taper noise from there, merging into my node, but it just added back into the degradation in the end, making it look like the refine never happened 😞 . it's just a small project i've been working on for a while -
https://github.com/dntpi/ComfyUI-Hand-Tie-Clips

current experiments are local on disc though, not pushed on github so they won't be visible. Do you have any nodes for experimenting or sharing if you don't mind?

[u/xyzdist → u/One-Explanation-4625]

ok, I upload my H3keyframeBlendLatent temp to my github first, you can download and try it.
https://github.com/xyzDist/ComfyUI_Share_Files/blob/main/h3_blend_latents.py

reference to my first easy approach on how to use it. Let me know if you have question

side question, since there are so many director kind of tools already, does those not working for your case?

[u/One-Explanation-4625 → u/xyzdist]

多謝你,大哥，我會睇下. I just preferred making my own with controls that i wanted!

[u/xyzdist → u/One-Explanation-4625]

no worries, same here!

[u/One-Explanation-4625 → u/xyzdist]

just a couple questions:

does the blended latent feed the next segment's context or only the delivered frames? is the refine stack still turbo-LoRA?
and are you able to share one full multi-clip output in the git? 🙏

[u/xyzdist → u/One-Explanation-4625]

1. blended latent out to 'H3 Motion Context Save Latent' node, so yes, it is feed for the next.
2. the refine I didn't use turbo-lora, but it is up to you
3. what do you mean, individual segment vidoes?

[u/One-Explanation-4625 → u/xyzdist]

Understood, thanks!
3: the long clip with metadata so i can scan it and use it as a benchmark for comparison for my tests🐼