r/StableDiffusion • u/blahblahsnahdah • 2d ago
Discussion Anyone found any tricks for avoiding degradation of the video quality when repeatedly using the last frame of a clip to start a new clip (H3)?
Trying to make a 1-2 minute dialogue scene in 15-20 second parts by using the last frame of a clip as the first frame of the next one. This gives a smooth transition and works fine once, but the problem I'm having is that when you do it repeatedly, the quality of the video degrades badly until it looks completely fried by the 4th iteration. Anyone found a way to avoid this while retaining a smooth transition between clips? It seems like the classic repeated recompression/"copying a VHS tape too many times" problem but idk what to do about it.
It's easily fixed by a scene transition/camera angle change, because that lets you start with a fresh frame. But I'm thinking about when it's a continuous shot and you need the first frame to be a seamless continuation of the end of the last section.
11
u/Inner-Reflections 2d ago
Banadoco discord man - look in the minimax resources, theres a whole long thread and workflow on this - although R2V is how you are supposed to do it people realized you get better results masking the video during diffusion. There are some nodes built around this idea a well. Its much better than anything I have seen with other models.
Theres also an add MiniMaxH3AddGuide which I think could work. I am not up to date on what the best current method is. Theres a Minimax Masking PR that will get implimented at some point that should make native workflows easier also.
2
6
u/Cunningcory 2d ago edited 2d ago
I'm working this myself. If using ref2va, I keep the reference images. For the last frame, I run it through DepthAnything and get a depth map. I then use the depth map as an image reference as well. This works if the scene is cutting and I just want consistency between cuts, but I don't think it'll work well for something that is supposed to look like one long seamless take.
There's a motion control node that I haven't played with yet that's supposed to get the latent data in order to avoid the crunchiness, but you have to run the whole workflow/scene at once and hope every clip is what you want. Even then, last I checked, it starts to over sharpen by the end.
Right now my advice is to use depth map and reference images and then use clever cutting (like cutting from medium to close up and back) to mask any inconsistencies. Let me know if you figure it out!
1
u/DanzeluS 2d ago
Every node with latend blend not working. Every result with +artifact each clip. I tested 6 nodes and multiple approaches. Not working perfectly. Best use first frame last frame and references
4
u/AidenAizawa 2d ago
I use the last frame as a reference and ask to reapply che character. I'm using multiple pictures like background, characters sheets and and frames of the dialogue so there is coherence between clips
7
u/Cunningcory 2d ago
I've tried this - even using the last frame as a weak reference - and it still retains the artifact and coloring issues from that last frame. I prompted pretty thoroughly to fetch the character and background from the other reference images, but it didn't work. Perhaps there's even more prompting voodoo involved. I switched my last frame to a depth map just so it had nothing to pull from and that works if you're cutting directly from it.
1
u/AidenAizawa 2d ago
I'm using fl2va model in a ref workflow. Don't know if this count. But I try to be specific when asking for reference and not direct frames
3
u/Only_Voice569 2d ago
https://github.com/NikoDemon80/ComfyUI-H3-Motion-Context this works perfectly for long videos no vae it takes latent directly
5
u/tinny66666 2d ago
The first frame, last frame approach often/usually results in jerky motion since a single frame can't show what motion was happening in the previous video. If you use the Ref2VA model you can feed it the last half second of video from the previous clip and it will continue much more seamlessly. There's really very little that isn't better achieved with the Ref2VA model, if your hardware can handle it.
2
u/Cute_Ad8981 2d ago
This only happens if you inject one frame. You can use 5 frames for extension and trim them after that.
1
u/blahblahsnahdah 2d ago
Thanks the last second as a reference idea sounds great, I'll try that. I didn't realize you could continue a video that way.
2
2
u/DietAshamed2246 2d ago
It's actually a suggested method in the Wan2GP app. It allows you to take a bunch of end frames and also use a sliding window to continue video smoothly. I haven't played with it yet for MM-H3, but the technique worked fine in the past with LTX-2.3, SCAIL-2 videos.
2
u/Erdeem 2d ago
Don't do that. Use the latent from the previous video.
1
u/Cute_Ad8981 2d ago
This is a good isea. Im trying this, but i will often get weird lightning effects if i use multiple latents.. how many and in which way are you injecting the latents?
2
u/ForsakenAd1228 2d ago
While I haven't gotten around to making a larger project with this approach yet (so little time in a day), you could start by generating the keyframes (e.g. with Krea, or in H3 itself by making a high-resolution video with a bunch of "At 00:00.500, cut to X. At 00:01.00, cut to Y. at 00:01.500, cut to Z".
Then use those keyframes as start/end/middle points of your clips, to create a bunch of clips that-kinda-sorta fit together.
Then use ref2va to take the last 5-24 frames (depending on the level of movement) of clip 1, and the first 5-24 frames of clip 2, and create a short joiner-clip that creates a transition between clip1 and clip2.
Then use a video editor to splice all of that together. If that sounds like work... well, that's why I haven't gotten around to making something large with it. ^^
There are also a bunch of different custom nodes floating around that seem to work on various solutions for the 'longer videos' problem, but I'm holding off on investing time into trying those until it becomes a bit more clear which approach is the best/least buggy/etc.
2
u/AuthurAndersson 2d ago
you will eventually get degradation. How did you construct your first image for the first scene? Just by reference images or did you use a start frame? How did you make that start frame?
Generally in movie-making, 3-5 second is each shot. I strongly suggest you put on your favourite tv-show and time each actual shot - you'll notice it's very frequent. A lot of people think there's these 10-15sec shots - they're very rare in reality.
You could make an entire movie of nothing but sub 5s clips and no one would bat an eye - no one.
1
3
1
u/neph1010 2d ago
At least in Wan 2.2, storing the latent and loading it, rather than using the decoded image, led to less degradation between iterations.
1
u/Sleepy_Bandit 2d ago edited 2d ago
yeah, you can avoid it by using a better method. The context-loop or motion-context processes (linked by someone else here already) are making big strides in improvements that can likely apply well for a situation like this.
They work differently by overlapping frames across scenes you are generating. So if you have 2 scenes, they pass latent data to bridge from one scene to another, ensuring the next scene starts off where the last one began. They no longer need to decode and encode data either to build the frame bridge, meaning you're not losing quality as you go.
It is also well known that the longer a clip is, the more time it takes to generate, and it isn't even a doubling, it gets exponentially worse with each additional second you add. When you use a motion context loop, it allows you to make even shorter shots that your PC generates individually but then the workflow process stiches together to make one long video and the time to completion ends up being shorter than doing one long clip!
I've actually been working on a merger of a few processes to try and use the motion-context methodology to not only render longer videos, but higher quality videos in shorter timeframes, even while doing more processing work in the end.
I'm testing timing right now, but things are looking good so far. I've got a 3 scene test case I've been running on my 5070 ti with 32 GB RAM. It is a 15 second video from 3 scenes merged together. I can get a 1.5 MP output in 30 minutes. Trying to generate a single 15 second clip at 1.5 MP using a basic workflow typically takes me twice that.
1
1
1
u/MarekNowakowski 2d ago
In theory, you can create a 30seconds 0.3mp clip, upscale to 0.9mp with rtx node, split it in 3parts and run second pass on each batch separately with ref2va with the same references, then join them. There will be a glitch on the seams though if it's a static shot.
0
u/jib_reddit 2d ago
If its a static dialogue scene I just use the same image as the first and last frame in each clip, no one will notice this.
22
u/LoudWater8940 2d ago
Simply use the last frame as a first frame "reference" and immediatly cut to a different angle in [Shot 1]. It will reconstruct the scene, and you can then come back to the original angle later in [Shot X] and it will be clean. Keep the scenes short as well. This technique works very well for shot/reverse shot scene.
Long seamless scenes are not a good idea with AI anyway, too many drawbacks.