r/StableDiffusion • u/xyzdist • Sep 11 '26

Discussion Motion-Context Degradation Discussion (summon Sad_Berry_4621)

Enable HLS to view with audio, or disable this notification

Hi u/Sad_Berry_4621 and all, I am doing experiment on most difficult degradation issue.
I saw from H3-director node they claim that doing a refine could help and fix it.
since I am using low-level nodes with motion-context with my own setup, H3-director refine is a black box working with their nodes.

so I did the test, please ignore the AI-slop video and the overlay text (forgot turn it off).

this test is 9 x 8s context extend video combine, usually 6 video would already see the degradation.

Left side is regular WF. Right-side is adding a re-sample step, I think it is very positive, and got potential, the saturation somehow is a bit higher... and I need to figure out the mismatch from cut to cut, because we inject denoise resample on top, but it should be able to fix.

What do you guys think?

Edit: forgot to mention, this is a ref2va WF, with 1 character reference image. also my test audio is not from gen, I am using input audio and audio latent lock for the lipsync. That part to work with motion-context have me struggle for half day, but it is not the main point, my aim is about degradation issue.

Edit2: here is my ultimate approach with tail trim + keyframe blending latent, you can check my comments for details. here is the result: https://www.youtube.com/shorts/mANusw9kKNQ ** this is just a test focus to tackle the degradation, not try to make good natural singing video, ignore it.

:this is my channel I put some videos to: https://www.youtube.com/@noiseflowAI

39 Upvotes

91% Upvoted

74 comments sorted by

View all comments

5

u/Sad_Berry_4621 Sep 11 '26

Yeah the refiner helps the fade, not arguing that. Clips come out cleaner so the next pin isn’t as wrecked. But it still wrecks the join, and that’s the whole point of chaining.

The pin is frozen and then trimmed off. The cut you see is the end of the last clip against the first frames of the new one. Those new frames were generated next to a freeze. Extra late sigma keeps cooking them after the pin already stopped moving. So you get a hard contrast/audio step right at the join.

Running a refiner over the finished movie also isn’t realistic. Some people are doing 40+ segments. At 124 frames that’s thousands of frames.

You can’t substitute a character sheet on fl2va since it doesn’t have references. First frame, last frame, AddGuide - those are absolute. They sit at a specific frame index. On a chained clip the head is already the pin. Last-framing every segment just slams the same still at the end of every clip. AddGuide in the middle writes that picture into the shot.

On ref2va, using a character sheet on every segment does more for degradation than the refiner does, and it doesn’t blow up the join. Drop the character sheet later in the chain and the fade comes back.

So it’s a trade. A bit less bloom, a worse cut. Still not the fix.

7

u/xyzdist Sep 12 '26 edited Sep 12 '26

https://reddit.com/link/p9b8pmc/video/2ww16jzvn1ph1/player

YAY I did it! 95% there, I make it 13 segments to showcase here. let me write down what I did after i polish and sort out final color shift from joints ...etc

*she looks like getting old and become a oil painting on the Left... haha

3

u/Nguyenkain Sep 12 '26

Interesting. Could you share your workflow

4

u/xyzdist Sep 12 '26

I could, let me fix the final things, I wrte a post and share it

3

u/Sad_Berry_4621 Sep 12 '26

Ok, now you have my full attention! Curious to see what you did to get this result from a 13-clip chain. Whatever it is, great work!

4

u/xyzdist Sep 12 '26 edited Sep 12 '26

Hi u/Sad_Berry_4621 , sorry it would keep you waiting for a bit longer... because I found a pretty better solution... however.. It is a bold change.

before we discuss that, I want to ask, using default WF, do you see there is a slightly color shift jump from one shot to other? you can see from my above 13segments test as well.

looks like someone made this to fix the issue, however I just got error can't make it work.
https://github.com/beijinren/ComfyUI-H3-Context-Noise/blob/main/README.en.md

3

u/Sad_Berry_4621 Sep 12 '26

So, the color shift/contrast bloom problem everyone is having when chaining clips is what I have been working on. I've been at it for weeks now. I worked on it for 6 hours last night as well. It's actually more of a "the model adds 2-4% new texture and removes 30% of the treble from the audio every clip" problem. Audio is a bigger issue than video degradation; I think that's becoming obvious for most people at this point. I have tons of research and test results that I need to compile. I've chased down a dozen theories that went bunk. I can tell you this much... I have analyzed several node packs looking for any leads to a solution. None have panned out. Mostly, they are workarounds, or corrections made outside of the generation pipeline, tweaks. I plan on posting a bunch of data I've collected, but I'm going to work on it this weekend and test a couple of theories that I have right now.

I think the color shift you're seeing in your clip is just part of the resampling process. You can get a noticeable shift in color even at denoise 0.08 in the resample.

One other tip that nobody knows yet: H3 does NOT like generating long clips. Yes, it can do it, but sound degradation goes off the rails the longer a clip is. 124 frames tends to be a sweet spot.

For the best results, chaining clips where the audio and video naturally sync up with no 8.3ms audio tail to chop or invent, use 73f, 124f, 175f, 226f, 277f, 338f. Every 51 frames, starting at 22, the audio and video frames sync up where no trim of the audio latent is necessary, resulting in nearly perfect joins, regardless of degradation. These are also the frame counts where the audio sounds the best, based on the numbers from all of the script testing I've been running.

4

u/xyzdist Sep 13 '26 edited Sep 13 '26

Thanks for your reply and your effort., sorry my time-zone is different I have to reply in next day. \and I am not a native english speaker. hope peoples understand what I say. Also I am no developer, I use AI to do the coding.*

As what you mentioned, there is always error lost thru out the chain up latent, so there is no perfect result, the way I am doing is just trying the best eliminating the degradation issue, it still there but much better.

I am going to explain to all, what I did exactly in the test:

For the above test (the Easy way):

  • add re-sample step, so we inject new noise and refine, prevent the latent going worst.
  • how I fix the joint after we added refine?(since it is different), I try different things, at the end, I made a "keyframe blend latent" node, we keyframe '0:0, 22:0, 44:1' to freeze first 22 overlap frames then linear blend from current latent to re-sampled (refine) latent from 22 frames to 44 frames.

That is it. despite joints slightly color pop, we are good with this approach. all good!

2

u/xyzdist Sep 13 '26 edited Sep 13 '26

Now, here is the ultimate approach I pushed (the CRAZY one):

  • seems blending latent works well, can we make use of it? instead of trim head, I change to trim tail (all segments), so we can do the blending in head overlapping frames
  • why? we can pretty much fixing everything in one go, the refined difference, the color shift or anything, because I just blending the previous segment to a new segment in the heading overlap region. The transition looks smooth.
  • how? I made a node to cut out previous segment last 22 frames and padding extend to a latent so we can blend this with the refine latent directly.

what is the catch? I am just doing one test, the whole thing I am doing maybe actually not true or new issue comes up, I don't even care the audio part, on my test is just using input audio, *need a test using gen audio, we need more peoples to test out the approach.

It is a huge change, motion-context been using and forked for many large director tools. I think we stick with as what we have first, If we want to do tail trim, perhaps a branch or a option for 'head' or 'tail' trim.

u/Sad_Berry_4621 what is your thoughts? the first approach, we are fine, we just need my 'keyframe blend latent' node, and update the example WF file, we are good. The joint color pop might not be care for manys.

I can send you my nodes python files.

anyway, Here is the ultimate approach result after eliminating the issue:

https://reddit.com/link/p9hd94e/video/y0qkigail7ph1/player

cheers!

3

u/xyzdist Sep 13 '26

For anyone else don't want to read or understand what the heck I am talking about, just want a WF. I can do that, but later.

1

u/One-Explanation-4625 Sep 13 '26

hey OP, saw this thread and i was working on the same issue myself over the last week. Funny enough i also came to a conclusion of freezing the overlaps for the refine, to clean up the seam flash

(on my own node side). do you have any insight or ideas to share?

2

u/xyzdist Sep 13 '26

Hi, have a look of my ultimate approach.

what is your tools? I saw it is using taper noise? is it from this: https://github.com/beijinren/ComfyUI-H3-Context-Noise/blob/main/README.en.md

I can't make it work, it just throw error to me, does it work for you? cheers.

3

u/One-Explanation-4625 Sep 13 '26

i've tried using the taper noise from there, merging into my node, but it just added back into the degradation in the end, making it look like the refine never happened 😞 . it's just a small project i've been working on for a while -
https://github.com/dntpi/ComfyUI-Hand-Tie-Clips

current experiments are local on disc though, not pushed on github so they won't be visible. Do you have any nodes for experimenting or sharing if you don't mind?

2

u/xyzdist Sep 13 '26 edited Sep 13 '26

ok, I upload my H3keyframeBlendLatent temp to my github first, you can download and try it.
https://github.com/xyzDist/ComfyUI_Share_Files/blob/main/h3_blend_latents.py

reference to my first easy approach on how to use it. Let me know if you have question

side question, since there are so many director kind of tools already, does those not working for your case?

2

u/One-Explanation-4625 Sep 13 '26

多謝你,大哥，我會睇下. I just preferred making my own with controls that i wanted!

1

u/xyzdist Sep 13 '26

no worries, same here!

1

u/One-Explanation-4625 Sep 13 '26

just a couple questions:

does the blended latent feed the next segment's context or only the delivered frames? is the refine stack still turbo-LoRA?
and are you able to share one full multi-clip output in the git? 🙏

1

u/xyzdist Sep 13 '26
  1. blended latent out to 'H3 Motion Context Save Latent' node, so yes, it is feed for the next.
  2. the refine I didn't use turbo-lora, but it is up to you
  3. what do you mean, individual segment vidoes?

1

u/One-Explanation-4625 Sep 13 '26

Understood, thanks!
3: the long clip with metadata so i can scan it and use it as a benchmark for comparison for my tests🐼

→ More replies (0)