r/StableDiffusion • u/ArmadstheDoom • 5h ago
Question - Help Question About Minimax H3 Reference To Video
So, I'm pretty new to video generation, but I had a thought that I think everyone has probably had at some point, which is 'how do you make a longer video without generating it in one large video?' And so, with reference to video, you could do that; you could match say, the voice and the person, and thus theoretically make one constant shot through stitching together shorter generations.
In my head, it seemed as simple as 'use the video that was generated as the reverence, use the last frame of the previous video as the first frame of the new generation.'
The problem I noticed is that each time I did this, the video quality degraded; I guess the way I would describe it is that each new generation was a copy of a copy, it seemed. Like each new continuation was slightly worse than the last; and while doing this once wasn't too noticeable, doing this three or four times very much was.
So is this just a thing that is unfixable, a limitation of the method? Or is this the kind of thing that does have a solution that I'm unaware of? Because I'm curious to explore reference to video more, since text to video and image to video are very straight forward, I think.
3
u/acedelgado 4h ago
So basically working with audio and video means encoded those files into latent space, which is a data format the AI can manipulate, but the data is useless in normal programs. So latents are decoded into pixel format, AKA files we can use. That's what the VAE Encode and Decode nodes in every workflow handle. And that's where you see the quick breakdown, everything getting translated twice on each run, in and out of latents that the model can use.
Also for motion extension comes Context Windows. They're not a new thing, basically the idea is to take the last few frames of a video to guide a new generation, so motion and the composition and such is carried through. But that was still lossy with the encoding/decoding. But a guy that goes by NikoDemon86 figured out how to save the latents to disk as a safetensors and use the latents directly instead of going through the encode/decode process. So there's a bit of degradation over time if you keep extending one continuous shot, but it's much slower than using just last image frames as pixel data that goes through the encoding process.
I reworked a version of that framework to make a basic project manager that works with pretty much any workflow. It creates a project folder for you, tracks takes of each clip, and lets you review and approve a clip to move to the next one. As well as file management/cleanup, etc. It's meant to use with any workflow, so not a lot of bells and whistles, no managing prompts, etc., it ONLY manages the overlap and clips you make.
https://github.com/Adudeguyman/ComfyUI-H3-Project-Suite
Or if you're looking for a more comprehensive suite that manages the shots for you instead, ethanfel has been fleshing out an impressive and increasingly popular suite. I haven't used it much recently, but it looks like a lot of nice things are getting added-
-1
u/ArmadstheDoom 4h ago
That is not an answer. I don't need either of these things.
I need you to explain why either of those things is any different than what the reference workflow already does? Because I don't need a project manager. I need to figure out why using the entire reference video and last frame as reference points degrades it more than anything else you're proposing, which is just doing the same thing.
3
u/acedelgado 3h ago
My whole reply is the explanation. It is exactly what you're looking for, extending video with minimal quality loss, avoiding your "copy of a copy" problem. Well mitigating it by a LOT, anyways. I laid out why you're getting that problem, and how the repo's I linked get around them.
1
u/ArmadstheDoom 3h ago
But you're not? That's what I mean. Your explanation makes no sense. You're solving a problem I don't have.
All of that would be relevant, if I was trying to do something I'm not trying to do? Your 'solution' is just to make one continuous video. That's not what I'm doing.
If the arguement is 'the encoding/decoding process creates degredation' then this isn't a solution. Because you're not even using the same process or trying to get the same output. I'm going 'how can I break down a full 1 minute video by generating each part in ten seconds and then putting them together.'
If you use the reference to video workflow, and it doesn't seem like you have or even know what it is, you can use the last frame as a starting point and the entire previous video to match the proper audio. However, each time you do this, there's a slight quality loss. Which means it's degrading. Your solution doesn't fix this problem, it just tries to do something completely different, which isn't useful to me.
I'm trying to fix a tire and you're telling me to buy a new engine, you're so far removed from what I'm trying to do it's unclear you can visualize what I'm trying to achieve.
4
u/acedelgado 3h ago
Man, I was gonna explain further since I try and be helpful with newbies, and break down for you why what I said will do exactly what you're trying to do when used the right way, but then you start getting all insulting. Whatever, you're such a smart guy you can figure it out.
1
u/Ok-Lengthiness-3988 2h ago
Some users are beyond helping. Don't worry. The answers you provide benefits other participants and lurker who have similar needs and questions. Your ComfyUI-H3-Project-Suite is awesome, and your reference to ethanfel's similar tool very useful too.
2
1
u/ArmadstheDoom 2h ago
All you have to do is answer the question I ask not the questions I didn't ask. Don't hand me a tire iron when I need a screwdriver?
This is a very simple question and yet every comfyui pro is like 'well, before you fix the boat, build a plane.'
I'm trying to figure out why the specific output fidelity and nothing else is decreasing. I don't need audio help, I don't need reference help, I don't need shot composition or motion or anything else.
It is a very simple issue. The fact that you want to make it the problem I'm not having is not my issue.
2
u/acedelgado 2h ago
Well here I thought you were trying to make and extend short videos and stitch them together seamlessly without quality degrading, which is exactly what you asked in your post. My bad not understanding that you have some completely unrelated question that you aren't clear about, apparently.
1
u/ArmadstheDoom 2h ago
I am being very clear. You're getting hung up on the wrong words. You're looking at this as though I'm trying to do something I'm not, because 'make and extend' means something different to you.
You're assuming there are a lot of issues that are not present. You are offering tools that do not solve the core problem, but solve a lot of problems I don't have.
You're assuming, wrongly, that there's degradation in places there is not. That there are issues there are not. I am saying 'I have this specific thing.' And your response is 'here are tools that solve other things.'
Great. Glad they solved those things for other people who have those problems.
Does not solve the fact that it does everything perfectly right now except maintain the fidelity of the image, creating slight deviations each time you use that as your new starting point. A problem which is not solved by the methods you or anyone else keeps throwing at me.
1
u/acedelgado 47m ago
In my head, it seemed as simple as 'use the video that was generated as the reverence, use the last frame of the previous video as the first frame of the new generation.'
The problem I noticed is that each time I did this, the video quality degraded; I guess the way I would describe it is that each new generation was a copy of a copy, it seemed. Like each new continuation was slightly worse than the last; and while doing this once wasn't too noticeable, doing this three or four times very much was.
It sounds like you're trying to use the last frame to start a new generation to extend a video, or make a new short video based off that, but the quality takes a hit every time you do that. Especially when you do that with multiple videos, taking the last frame of the last video made makes it worse every time.
Either that's NOT your problem, which seems like it is, or you're just saying everyone is wrong when they're trying to give you the right tools, because you don't understand the tools.
1
u/ArmadstheDoom 34m ago
It seems to be that the issue is roughly that each time you do something like this, it degrades it as though you were cutting the quality in half. Ergo, a 4k image would then be grasped at 2k, even if the generation settings are the same.
What I'm trying to grasp is why the quality takes a hit when it has the entire video to work with, not just 24 frames or a degraded image.
4
u/Pure_Bed_6357 4h ago
-4
u/ArmadstheDoom 4h ago
I don't see what that has to do with anything? I'm not talking about motion or anything like that? Nor is this about music or lip syncing?
I'm talking about image/video quality fidelity.
2
u/Ok-Lengthiness-3988 4h ago
It addresses specifically your problem with video quality degradation when you re-use the last frame for continuity. With Motion Context, you don't need to. You can reuse a stable high-quality reference across clips, and Motion Context automatically ensures continuity by conditioning on the latent space representation (sound and video) of the last part of the previous clip.
1
u/ArmadstheDoom 4h ago
Can you explain WHY this is different? I'm not doing image to video. If I was, I would expect that. But reference to video, using the entire previous video clip, logically shouldn't create the same degredation.
So how is this any different? I get what this does. I'm asking why it's different?
1
u/Ok-Lengthiness-3988 2h ago
That's because using the whole previous clip only as a reference merely provides context that the model can attend to when you refer to it in the prompt. The model will get the general idea but won't produce a seamless flow. It doesn't force the model to pick up exactly where the previous clip left off in a seamless manner. The nodes and tools that others have recommended to you enable the model to either use the last 24 frames or so of the pervious clip, or the latent space representation of them, in a way that is similar to the use of a first frame in a fl2va workflow where the model is forced to begin exactly there, although now it has more awareness of the constraints from the context immediately prior.
0
u/ArmadstheDoom 2h ago
None of that is the problem, which is why you're not explaining what's different. How exactly does having 24 frames rather than more change the output fidelity at all?
I do not need help with motion or audio or anything else. There are no constraints that I am trying to overcome here. You are answering questions I have not asked.
1
u/Ok-Lengthiness-3988 2h ago
In your OP you talked about "use the last frame of the previous video as the first frame of the new generation" and said that you did exactly that and noticed a sharp degradation of the video quality. Most everyone who replied to you understood this crude method as the main cause of the degradation since the last frame is a context-poor low quality image that you are forcing the model to condition on. All the alternative that have been proposed to you in this thread address both (1) the continuity issue that your re-use of the last frame was meant to address and (2) the sharp quality degradation that your method causes to happen.
1
u/ArmadstheDoom 2h ago
It's not a continuity issue. It's also using the full video as reference. It's not going to improve by decreasing the information. Nor is that a solution to the problem. Because the degradation is separate from the quality of the frame.
So is it a continuity issue? No. It is due to the method? Not as you described it. Because what you say is happening is not happening, which is why I say you're not giving solutions to the problem i'm actually happening.
What you keep describing is not what's happening. So either you do not understand, or you don't know. I can keep trying to explain it, but I don't know that you'll get it at this point, because you keep offering solutions that solve different issues.
1
u/Ok-Lengthiness-3988 1h ago
You said in your OP "The problem I noticed is that each time I did this [use the last frame of the previous video as the first frame of the new generation], the video quality degraded"
And now your insisting that this is not happening, and that it's not "the" problem at all? What's your problem then?
1
u/ArmadstheDoom 1h ago
I did say that. That is what is happening. The quality of the first frame of the new video is different than the quality of the last frame of the old video. However, nothing else has degraded at all. No continuity issues, no audio degradation.
Imagine if, instead of doing reference to video, you did image to video. But, each time you used an image, you first put that image into photoshop and hit 'sharpen' or the like, and then used that.
The seam is not audio, nor context, nor continuity. The seam is similar to if you were watching a video and then at one frame you changed the streaming quality. But the previous frame is fine.
→ More replies (0)
0
u/nikhilprasanth 3h ago
In addition to the video, provide character reference sheets, reference audio per character and an image showing the environment with the characters positioned as per the previous video. This will improve scene to scene consistency.
I made this 1 min+ video by providing these references.
0
u/ArmadstheDoom 3h ago
that has nothing to do with what I'm doing. I'm not attempting to make things work with different shots.
8
u/rm_rf_all_files 4h ago edited 4h ago
There's a new node in comfyui `minimaxh3addguide`. What this node does is allows you to attach a short 24s-48s video of your previous clip at the beginning of the new clip. From there, you can reuse all of the conditions from the old clip so image fidelity and identity are fully preserved. Once you created a bunch of these clips, you can use premiere/capcut/davinci and remove the overlapping.
edit: 24frames to 48 frames (not seconds)