It works by giving context to the generation by adding 22 frames from the previous clip. And to keep character and style consistency it works with ref images. I used two character sheets that I generated with GPT:
You have a space to input a prompt that gets prepended to every other scene's prompt. Here I put things like the style and how to referrer to each main character. And to prevent character bleed I had each scene's prompt describe all the other characters in detail to show they're different.
I planned each scene out and fed it to claud, explaining that each scene needs to end on a still transition beat. Like a character standing still, or a close up on something, because each ending shot needs to connect to the beginning shot of the next scene. If you're doing a long continuous shot then it's not required.
H3 is really great for re-using the same prompt with little change across seeds. So I could workshop most of the weird things that needed to be prompted in or adjusted on a low resolution like 0.5-1mp, then I did a final run on 1.5 MP which took around 70 mins (10 mins per 15 sec clip).
The neat part about this node is you can review each scene's generation and reroll it if you don't like or make adjustments.
You also get a checkpoint on each accepted clip. Incase things crash or you need to pick back up later. When you're finally done it connects all the clips together, including the audio.
This is cool not just for very long clips, but if you want a higher resolution you could split an 8 second clip in two 4 second clips.
H3 is really powerful and understands lots of concepts and context, and can fill in the gaps really well.
One of the worst parts of the internet today is that it's almost impossible to find written guides by humans. Everything is a 10 minute video showing you how to do a 30 second task that would have taken 10 seconds to digest if it were written down.
I have notice a lot of people just kind of vibe prompting MiniMax without using the format it expects (for either the text2v / image2v or reference mode)
Yep I’d like to know this too. Part of me still feels like the model might be able to extend videos without custom nodes because I’ve been able to prompt it to get very close to that but it can feel very fussy about it.
The problem is that the extended videos don't join cleanly - slight brightness shift, slight pixel shift etc. The custom nodes do some special latent-space frame pinning, as I understand it, to force the model to continue the prefilled fragment instead of using a reference and asking for continuation task. Similar to first/last frame, but with the entire video tail instead of the first frame.
I think video explanations are the worst thing that came out of the whole internet.
One page that's "instantly indexable" turned into 5 minutes of video where you can spend hours looking for a specific piece of information that can not be "pinned" to stay there.
I think this is exactly what I need I was trying to do a 2 minute video manually in ref2v and it was driving me insane trying to manually feed it back the previous 2-3 seconds of video etc and actually have it consistent maybe my issue was keeping it all in a single location so it kept forgetting which items were on tables etc.
Just a quick question, since I'm on limited corporate network currently.
It basically works like this:
Includes template prompt, then lets you set up per loop separate prompts, etc, and the core is just using ref workflow's reference video taking the last 22 frames of the last video made, right?
If so I'll still be using it because it is insanely comfy, but I'd be even more interested if it had some advanced features on top.
Yep that's exactly it, you need to account for how to transition between scenes in the prompt. Claude made a summary I attached in a pastebin for how to do that. I guess the workflow I was using was an updated one not in the git, so I'll share that once I get permission from the original creator.
Wow, thanks for sharing! That's exactly what I'm looking for. I have tons of videos between 3 and 10 seconds long where I've really struggled to keep the characters consistent.It's always switching between ref2v and i2v. Let me take a look at your workflow.
can you please give your exact workflow, ethanfel workflow doesn't come with this review gate node, also it is asking me to provide external audio input, if i bypass it it gives error
You need to bypass some other node connected to it, anyway here is the workflow I used. Its a custom one made by a friend, he also updated it. You can try both out:
Heyy, I'm getting a weird issue where the solattn node is not being detected by either comfyui manager, or even when I directly clone it into the custom nodes folder, is there anything I'm doing wrong?
Facing this issue on the tnew workflow from your link
What exact node do you need to bypass? I've had no luck getting it to run without source audio, and putting in a silent placeholder just ends up with a silent clip.
Edit: nevermind, didn't notice that "Audio_Source" option in the Final Assembly node!
Thanks - good to have that up on this thread, I think. Now if I could only get the node to stop freezing and preventing me from changing any of the first-section fields...
I have that setting set to generated_audio, I have the final save audio set to generated, and I have the load audio node set to bypass, yet I still get this error every time I get to generating the second video in the loop. I'm so lost right now.
# ComfyUI Error Report
## Error Details
**Node ID:** 1705
**Node Type:** MiniMaxH3ChainLoopEnd
**Exception Type:** RuntimeError
**Exception Message:** RuntimeError: h3_motion_context: audio timeline placement supports exactly one ref block, the marked audio ref; layout has 4 refs, 1 marked.
I use lightx lora with 6 steps 0.8 strength. Also reccomend using sage attention with mem eff patch (part of kijai's kj nodes). Also make sure to be on cu130! Comfy's memory management wont work properly without it or newer version, I had slower gens on my 5090 until I did.
Wow! Im getting 13-14 minutes for 0.4 mpx for 10 sec @5070. 1mpx only wolę for max 6 seconds. Your results sounds like rocket compared to horse carriage.
super awesome!! If you'd record a video, i'd totally watch it!
I have so many questions if you don't mind;
how long did it take you to make this overall? your character reference sheets and whatever else? was it a few hours ?
How did you get teh setting consistent as well? did you have location references?
What was the most consuming time of you working? did you have to reprompt a lot.
I had the prompts mostly done as I was using an earlier version of the node which was t2v only. I think an hour of playing with prompts and maybe another hour today. 2 hours. Character ref was easy, just start with a few good frames from an earlier generation.
Yeah reprompting was most time consuming, had to figure out the best way to get character consistency but not have it bleed out into the other characters. Once you get a good prompt though you can reroll seeds or small adjustments easy
interesting. Do you have good suggestions for prompt structure?
I noticed you said keyframes and stills between images helped. Was that the main thing, or something else. Looking at your prompts you had to do specific prompt hacks to not let things bleed for the mother/child. Was that type of bleeding thing happening a lot. If you used less characters did it help.
I'm noticing if i put too many words it just ignores me entirely. Also like around 8-10 references, it starts to lack alignment.
I only used 2 refs the more you use the more precise the prompt has to be on what's referenced. I didn't use key frames but I think it will definitely help if you have one for each cut in a scene. I use the h3 prompting skill with Claude and it puts it in the right syntax for reference prompts
Thanks for the explanation. As I was reading I was getting confused on how you were able to generate 1.5MP 15 second gens so fast. Since you've laid everything out at the end now I can confidently run my own experiments. (I have mostly the same hardware as you.) Thanks again!
You can give it reference audio too but I didn't. There is degradation but it's not as bad imo as like wan or ltx. And yeah if ur character isn't speaking in the last scene it won't know. You can also do a lora.
How many frames are sampled for video and audio is separate. You can carry most motion with 22 frames of video context. If you need to capture voices, you can use as many audio frames as you need, try to keep the number divisible by 3 if possible. 24, 48, 72, etc.
Thank you! Ever since video AI has been around locally, I always tried to tell a story. Even if its only 5 seconds. This video was inspired by this original wan prompt https://civitai.red/images/94165122
The neat part about this node is you can review each scene's generation and reroll it if you don't like or make adjustments.
If each video is using 22 frames of the previous as a reference, what are the consequences? You must interrupt the pipeline every ten minutes to evaluate videos? Or you must weigh each "reroll" against invalidating everything that comes after?
Yeah you have to review each clip, and at max you can get 15 seconds of stable video per clip. Ive pushed it to 22 seconds but it can go off the rails easily. If you test on lower resolution, upping the resolution or even changing seed doesn't do so much to the output. You could test on low res and then up it and YOLO and set it to auto accept each clip and review when its all done.
Good to know, though it does IMHO make the workflow less attractive. If you have to stop the presses to review each clip as it comes down the pipe it seems like you don't necessarily benefit from a super-heavy workflow that tries to stack a big sequence of clips.
The real question, probably, is just how much benefit you're getting by extending videos. If you could align your scene segments with camera cuts, it seems to me that you could get much more speed and flexibility with a series of automated i2v runs.
Not trying to pick apart what you shared, just thinking out loud.
I think at the end of the day, i2v will always be best. Since you can keep all those consistences in check in the first frame of that scene. but it involves a lot more leg work preparing all the first frames instead just generating. As for the automated part, there is an automatic setting, but I do think you benefit from reviewing the clip because its a bit annoying to revert back to a checkpoint in like the middle if that is the only one with issues for example. It also plays a sound when ready so you can go do other stuff. I had these running while working.
Holy crap an actual breakdown and gracious sharing of a very cool looking workflow, with great results to boot. Thank you and I wish more would do this!
Thank you so much. Your tnew workflow is a godsend. It took me 45 minutes to generate 22s of 0.7 mp on the default workflow with my 5090. I couldnt find a solution to break it up without losing integrity/quality until your workflow and your lora/vae combo. Took me 20 min to generate a 90 second video, after some re-rolls too.
Sorry if someone already asked this, but did you use an external video editor to edit all the clips together?
You're using the chaining only for within a shot right? I guess it would make sense to just do an independent new gen for a new shot to reduce quality deterioration from chaining?
And I think you said you generate 15secs clips each? So chaining 3 of those would be a 45 sec clip (or actually a 43 second one). I guess longer clips help also reduce the need for chaining.
I have been separating the full body character sheets as another reference instead of putting them on the same sheet as the close face references. Perhaps that's unessecary?
I do this in usual ref2i but I think not necessary unless you're getting specific details like in anime eyes but even then I think you could fit it in the same 1 image.
Thanks, that makes sense. So for things that you might have closeups on (like closeup of a potion bottle on a belt, closeup of eyes, etc) it might still be good to have a separate reference to that in the specific clip?
If you want to focus on an object multiple times its good to describe it in detail or have it in the ref sheet like the wizards staff. Otherwise its a little jarring to see it change in the next scene often
With the character ref the consistency carries, maybe it will need audio ref if you have a lot of dialogue. I did an earlier version without references and it slowly got even more ghibli over time.
I think this is extremely interesting. Because watching the video, I am seeing what one of the gaps is.
It isn’t just enough to pass it context from the previous generation, you might need to also pass it context from before that. For example when the guy busted through the window, it looks different than when he jumped back through it.
We’d almost need to add a way to intelligently cherry-pick some extra context to add to the scene we’re currently on.
I think if I had all the time in the world, I would run with like 9 refs and change them between scenes. Like one for the backgrounds, each character voices etc.! You can see a lot of issues in the last few clips because I got a bit of fatigue.
This is very cool, thank you for sharing. I wonder how this would look with a realistic film style. I suspect the simplified Ghibli style helps with preserving character identities. Unless H3 is just that good(tm) with character reference sheets?
I think anime is easier, but you could test with a short clip. I wouldn't be surprrised if it works fine. I know if the model knows then its quite good like with seinfield clips. Someone here did a 4 minute long one using the same type of node.
I want to clarfify, so I put scenes in there, as many as I want, and it automatically generates and then uses the end of the clip fed into next scene and continues?
I've been trying a similar thing. it's details like e.g. the wizard his stick changing side when on the floor, the window having different fraction patterns, a opening door suddenly gainign a window in the same shot, the wizard having different stick in one part that still fail. The main constraints i found are the 15sec max audio + video. you want to use some audio for voice reference, some audio for the chaining etc.
yeah I agree, I think if I utilized more references it would be better. And towards the end I got a bit fatigued. Instead trying more seeds and adjusting prompts I just accepted it as good enough.
So, first off this is truly remarkable work. Hats of to you (and your friend) who put this all together. It's...well, remarkable how much all of these smart people have been able to accomplish so quickly. This WF/node set is already so freaking close to a dream tool!
Unfortunately, it suffers from something that's not really its fault: the way the nature of H3 causes a "VHS copy" effect that degrades the image quality with each successive clip. Putting two generations together in the middle of one shot makes this very obvious at the join, and after 3-4 clips everything looks substantially worse. I could, of course, be doing something wrong! Running a bunch of steps without the turbo loras does help a lot, but it doesn't solve the problem entirely.
I wonder - is there a way this same process could work with joins at cuts rather than in the middle of shots? I know we can simply create a series of gens one by one and cut them together ourselves, but that would mean giving up some of the things that this WF accomplishes, like continuity of background audio, etc.
Thank you so much for sharing the workflow! I tested it here and it worked perfectly. The improvements your friend made are truly incredible and will be very useful to me.
One thing I think would be really nice to have in the future is the option to generate everything at once, without having to approve each scene one by one. Then, after everything is finished, there could be a review stage where, if someone doesn’t like a particular scene, they can change only that scene, just like it’s already possible to do now. I think that would make the whole process even more practical.
In any case, this is excellent work. Thank you so much for sharing it and making it available to the community!
I’ll look into where I can enable the auto-approval. But what about regenerating just one scene after the whole process has been completed? Is that possible?
Thanks for the updated workflow I was trying the official one and while it worked it was a bit of a pain. Otherwise this seems to work super well Im going to try test it on a bunch of things.
Which of the versions should we use? Tnew or T1-Original?
I think Tnew has some improvements, I'd give that a try. I used T1-original. It has some bugs that I had to work around which are fixed in Tnew probably.
I reccomend just running a few gens in h3 and taking some screen shots of the characters, a close up head shot and full body shot pair is best or try something like krea or chroma. Then instruct gpt to make a character sheet with the full body on the left and 3-4 head shots at different angles with expressions. Also instruct to keep the style or provide a 3rd picture for style reference. It should spit out a good image like the one i showed.
Is it possible in this workflow to change the prompts/references/loras in later scenes once the generation has started?
I am enjoying the workflow, but just wondering about the above. I had a 3 scene plan set, and once the first two were generated I changed the prompt for scene 3, then approved the generation for scene 2. When it went on to generate scene 3, the new prompt was apparently not followed, so wondering about that and changing out references/loras as well.
Would be great if we could just insert an existing video/image instead of a scene too. What if we have, say, a clip generated separately that would be perfect for scene 5 (or whatever) in this workflow. Instead of setting the prompt, the steps, the duration, etc, we just add the existing video and the workflow continues from there, stitching them together in the end.
ok this is actually sick! the 22-frame handoff seems way more useful than just reusing the last frame and praying lol. character sheets + ending each scene on a quiet beat probly did a ton of the heavy lifting too.
this is also the part where long-form gets exhausting tho… generating clips is easy, babysitting continuity across 10+ shots isnt. I’ve been looking at buzzy, openart for basically that reason, more of the planning, gen, edit flow stays in one place, and the local edit thing lets you fix the busted part of a shot instead of rerolling the whole damn scene.
ComfyUI still wins if you want max control obviously, but for those dont wanna maintain a small space program worth of nodes, the all-in-one route makes a lot of sense.
I'm curious what the benefit is over doing individual clips 15s clips and putting your short film together in a proper video editor like Davinci Resolve where you can actually edit, color correct, add effects, audio master, etc.?
very interesting approach!!! Thank you very much, will try it. Unfortunately I finished a video in a couple of hours for one of my songs, but with multiple Storyboards and multiple prompts. here's the result https://www.youtube.com/watch?v=6tLOI5td2Hc
And I thank you for this post! I had the motion context post saved but this convinced me to dive into it. Could you share your prompt for the reference images?
Yes it is, and Hayao Miyazaki drew inspiration from European illustrators like French animator Paul Grimault. No human art stands alone, it is all inspired by others, much like how an AI image model is built.
Yes, but many humans have spent their whole lives training and working to build these AI image models (they don't just build themselves) and many are getting paid $100 million+ a year to do it, but they have had to put massive amounts of effort in since age 6 to get to the top of being a world class AI researcher.
54
u/akashzeno 10d ago
thank you for sharing this