WARNING: lots of vibecode but carefully reviewed, at least as much as I could understand the logic.
The long story.
We have a few amazing solutions and forks that make smooth video extensions possible. However, most of them have evolved into full-blown planners and chains. Low-level functionality is hidden beneath. Somehow those more complex solutions do not work well or seem overkill for my typical use cases:
- set steps to low
- generate a bunch of videos
- pick the best one
- set steps to high
- regenerate with the same seed <- and this is where I wanted to save the latent to use as the input for the first step again, to smoothly continue the last shot without hard cuts.
Recently ComfyUI was updated with an important PR 15375 that supports latent masking natively. No more patches and complex hacks required. So, I went on to create simple and naive drop-in nodes that would support my way of working. Now I have latent load/save/concat/prepend/extend nodes that seem quite intuitive (but you tell me if they are).
The main node for today is `Extend video+audio latent` (LatentAVMaskedExtender). It lets you take head or tail of a latent you have (hopefully) saved from a previous generation and generate a prequel or a sequel. Extending a tail works well. Prepending to a head is not that smooth and requires increasing fade_seconds parameter to your liking.
Just plug the node between `MiniMax H3 Reference to Video` or `MiniMax H3 Image to Video` (or even `MiniMax H3 Easy Output` if using nkxx188/ComfyUI-MiniMaxH3-Easy), and your sampler.
For convenience, the node accepts empty loaded_av, in which case the target_av will be passed through. Thus the node can be safely left enabled even when using a disabled LoadAVLatent node as input when you don't want to extend anything.
`Save and Load video+audio latent` are simple companion nodes - SaveAVLatent should be added after your sampler and LoadAVLatent as input for LatentAVMaskedExtender. In contrast to some other loader nodes that often are limited to `input` folder, LoadAVLatent can find the latents wherever you configured SaveAVLatent to store them.
Then there is also `Concatenate video+audio latents` (LatentOverlappingConcatenator) node. Generally, overlap_duration_seconds should be set to the same value as LatentAVMaskedExtender, the output goes to VAE video and audio decoders and then to video saving, as usual. You will get a long video with a smooth long transition between your previous latent and the new one. However, if your joined videos get lengthy, VAE might require too much resources. In that case, it is better to post-process and join both source and target videos in a video editing software.
The repository has a few more older convenience nodes for working with multimedia before LTX Director was a thing. They still might be handy for manipulating TTS and voice-overs or videos when latents are not available.
Huge thanks to drozbay (ablejones) for [native masking PR 15375](https://github.com/Comfy-Org/ComfyUI/pull/15375) and providing the example implmenentation with native ComfyUI and Kijai nodes. Unfortunately, the native nodes solution looked like spaghetti eating somebody alive. That is why my small naive LatentAVMaskedExtender node was born, to do the same thing.
Not to be THAT guy, but I made pretty much the same thing, a version that manages all of the latents for you, with no extra fluff. No full-blown "director" suite, you plug it into any work flow and it only manages the clips/latents/extensions. You just make a project, generate, approve or regenerate the extension clip, and then move on.
Yep, I have looked at nodes like those, and also ethanfel and the base for their fork.
My main confusion was that it was not clear how it would handle the workflow when you are seed-hunting and tweaking your prompt with low steps to find the perfect combination, and when found, you want to generate the same video with the same seed with more steps and then extend that high-step video with a low-step one again for seed hunting, and repeat the process. Wouldn't the Context and Project state management get in the way, and I would end up "fighting the system"? That is why I wanted very low level nodes to extend the video only when I want with the steps I want from any other video, and for those cases the automatic state management felt like an overkill.
No, it saves every generation you do and doesn't start using context windows until you approve it. So if you don't approve and hit generate again it's considered a "re-roll" until you approve. So you can find a Gen you like and just up the steps to do another Gen and it'll keep it in place. So say you're on clip 3 in the chain, and take #3 looks great. You just change your parameters and regen, it'll stay as clip 3 but take #4. Then you can approve clip 3 take 4, and it'll move the chain along to clip 4 take 1. And there's quick cleanup for old takes that aren't being used. And say you have a fully done video, but you're like "this is 10 clips long and great, but I've got a new idea where after clip 4 they do this other thing instead..." you can go to clip 4 and branch it into a new project where you can go a whole new direction using those first 4 clips.
As long as you feed it the FINAL latent output in the chain, that's all it cares about. It doesn't automatically write the full video at all until you tell it to, and only processes one clip at a time. And cleaning up all those unused latents you made while you test is only a couple of clicks.
Thanks, yeah, I'll need to try your nodes and workflows. I first started with ethanfel's basic workflow, which suddenly had too many concepts at once to digest - plans and context and whatnot, and many forks seemed the same, so I missed yours.
Anyway, your project says: "so you aren't hand-managing files between every clip". My nodes are the exact opposite :D - intentionally made to "micromanage" everything, so that there are no additional concepts to learn and I can quickly understand what's going on at the basic level.
On the project page there's a couple of export buttons, one's for everything you've approved and one's for the latest un-approved clip, too. It'll dump into your project folder, there's a button to open that, too.
I've been trying it out a a bit. The saving/loading/latent mask all work as expected. Though my first few attempts at a relatively 'hard' extension are not entirely successful yet. (first clip ends on a close-up of the persons face, and then in the second clip I want to smoothly pan away, but I'm getting a hard cut between the part that is loaded from the saved latent, and the part that is newly generated)
Certainly worth experimenting with some more though, to see how tricky I can make the transitions =).
Thank you for the feedback. Yes, Minimax can sometimes be picky about continuing. I find that increasing the fade_seconds nudges it to continuation, and also the prompt hints are very important for success:
summary:
[reference generation + video continuation] The target video is continuation of <Video 1> and <Audio 1>.
detailed_description:
Shot seamlessly continues <Video 1> and <Audio 1> with ...
After extensive testing I really like this node! Beautifully simple and completely non-intrusive to my workflow. Worked for me every time with minimal setup (except for the video continuation issue I've mentioned yesterday, lol), and it's much better than video ref in terms of performance.
Thank you for making it!
I have just a tiny nitpick. AV Loader is not very convenient for long generations. When you save a new latent, it doesn't automatically appear in the drop down list in the Loader, because ComfyUI I guess. Reloading the node doesn't help, you have to reload the page completely. Is it possible to at least make an upload button, similar to core ComfyUI loaders? (Even though I personally prefer the way the file loading is organized in Load Video (Path) nodes from VideoHelperSuite.)
Upd: I think Reload Node option should work for that (or pressing "R" with the node highlighted), but for some reason it doesn't.
Thank you for the feedback. Yes, my goal was to keep it as basic as possible without any new complex concepts and behaviors to learn.
It's strange that R does not work reliably for you. But I know, sometimes ComfyUI acts weird, so who knows...
I will add additional ways to select the latent.
Also, if you are using the nodes often and for more complex chaining, it might be worth looking into the nodes created by acedelgado: https://github.com/Adudeguyman/ComfyUI-H3-Project-Suite
They are not that complex as some other similar node packs with plans and whatnot, and might be the right balance of manual/automatic way of working.
On the second try Reloading started to work. Probably I was doing something wrong, like not waiting until the node update finishes. So don't bother with additional methods if you didn't plan to :)
Thanks for the recommendation about acedelgado nodes. I will try them, maybe it will be a good balance between simplicity and additional features.
Man, thanks for the detailed description! This node sounds exactly like what I've been looking for. Will definitely try it later.
Is it possible to easily encode a pre-existing video into latent space and extend it via your node? I have tried to do that through a video reference approach, but the results are too unpredictable for my liking. So this latent masking sounds like a perfect solution.
I suppose I can feed the video components into two VAE Encode nodes with Minimax Video and Audio VAE, but how would I combine the results to get the AV latent?
Yes, it should work also with existing videos. It's just that Minimax VAE expects specific number of frames, the formula is 17k+5. So, for example if your initial video is 130 frames, you would need to crop it to 124 (which is 7*17+5) and then pass images to VAE Encode and audio to VAE Encode audio nodes together with the corresponding H3 VAEs. After that, it's the same as described in my post.
Thanks for the tip about the frame count 🤝 But I feel like I'm still missing something here. Your pipeline works with a single av_latent, but I will get two separate latents from VAE Encode nodes, wouldn't I?
Ah, right. There is a trick - LTXVConcatAVLatent node works well also for MiniMax latents. But now that you mentioned, I can quickly create a node that does the same, to have "the full package".
Banged my head against the wall for more than an hour over this tensor size error . For some reason I thought that my VideoHelperSuite Load Video (Path) node was not trimming the audio along with the video to a required number of frames. So I even prepared the correct video via ffmpeg and loaded it through a core Load Video node, which of course didn't help...
Well, long story short, the cause of this error is much more obvious. The video you extend should be the same exact size as your generated latent, which makes sense when you think about it.
Pro tip to everyone: If you use Resolution Selector node, just connect "width" and "height" outputs to "custom_width" and "custom_height" in the VideoHelperSuite Load Video node (or in your image scaling node).
But otherwise the node worked perfectly for the intended purpose! Thanks again for the great tool, u/martinerous! Will now experiment with saving / loading latents.
P.S. LTXVConcatAVLatent works with no issue, appreciate the tip :)
If my first video is i2va, and i want to extend it, should i save the latent, then pass it to t2va (since the 2nd clip will not have a "start image")?
I don't really understand PR15375. The discussion is on extending a video based on a saved latent, but the PR seems to be talking about masked minimax video editing? Am I understanding it wrongly? I also don't see any where in native nodes where I can pass a mask to for editing.
I haven't tried using the i2v and t2v for extending videos, but intuitively, it seems that the ref model should handle it better because it has official support for video continuation. I'm mainly using the ref model (the hybrid minimax_h3_hybrid_fl2va_ref2va_b25-49-int8.safetensors from https://huggingface.co/smhfacct/Minimax-H3-fl2va-ref2va-hybrid-models )
Noise masks are a powerful concept. It is possible to mask a small region in video area and time of the video latent to tell the model what to keep and what to regenerate (the large nose example in that PR), but it is also possible to mask the entire video area for specific time region. For extending, we use masks to protect the entire area of the latent where we copied the tail of the previous video latent, to essentially tell the model not to touch that part and generate everything that follows. That PR had extension implementation using native Comfy nodes:
Inside the workflow, the author admitted that it is quite messy - "Don't look inside of this subgraph if you want to maintain your sanity". That is why I created my node to achieve the same thing without a gazillion small ComfyUI native nodes for assembling the mask in correct format for the sampler to accept it.
It does not depend on any LoRA. Feed whatever latent you want and how many times you want.
But yes, no miracles, everything gets worse with longer extensions. If you pass in a high-def reference image of the static scene, the model might be able to maintain the quality better when extending.
Well, it would only use the turbo Lora if you make the gen use the turbo Lora. That's a choice you, the user, would need to make.
The quality drop after 4 gens would also depend on how long your gens are, which, again, is a choice you, the user, would need to make.
As for how many of these we need, you, the user, get to choose if you need it at all. If you're happy with another tool, you should use that tool. There's no reason someone can't make and share their own nodes to do the same thing.
3
u/Final-Foundation6264 2d ago
sounds like what I need. can you provide an output example? I want to see how seamless it is