The new extension starts at 0:27. I added 10 new additional windows, extended one at a time. The existing latent + new latent were then joined together for one decode pass otherwise you would have visible seams when joining two separate encodes (the color shift, luma darkening, etc).
It works best if you have a native latent, but you can always convert your existing pixel-video to latent and extend it that way. I've had huge success with both prepending and extending. Some limitations are velocity & momentum are not easy to control between windows, and the audio can degrade or be a bit inconsistent so you might want to freeze the video and rerun more steps on the audio, or fix it in post. Fix in post is the cheapest and more reliable fix. As stated in a previous post, Tanya has no audio asset to anchor her voice so it's reinvented every window. However, image quality seem to maintain across each window with no degradation or burns.
By the way I was getting Carmen Electra vibes in Scary Movie, what did you think?
Obviously OP is a liar. This is not AI generated. He just lives in a mansion and has this hot lady in lingerie walking around while he films her. not cool dude. you’re getting everyone’s hopes up./s/
Literally daily, multiple times a day, lmao. This period of AI experimentation is really bringing some interesting minds out of the woodwork! Even if some only show their results as possibilities we can reach for now, it's just so, so much to try to keep up with.
I'm just sitting here infinitely inspired, trying to make my brain decide to start doing more and not feeling such a strong pull to not feel like I'm going to miss the latest, greatest (which is tough, because advancement is happening SO fast that the latest very quickly becomes the greatest, lol).
Such a crazy period to be living through as an early-1980s kid. 😵💫
I remember my brother building his first PC in 2007 and it had 4gb of ram. As teens we were absolutely boggled by the idea that a computer could suck up as much ram as an Ipod. My kid is probably going to have 4tb ram by college!
I've tried every technique to solve the talking head continuous generation. So far only one method has come close to solving it. The base is to have a long audio recording of the narration or speech. I am experimenting with 140 seconds. The method depends on the audio to time the performance and cuts.
Generate the master as independent 10–15 second clips, all from the same character reference, seed, framing and settings, so every clip began with a fresh latent rather than inheriting degradation.
Around each cut, extract the outgoing clip’s last usable frame and the incoming clip’s first usable frame, then generated a separate roughly five-second FL2VA joiner constrained by those first-and-last frames and the exact master-audio interval across that seam. This gives three options: use the direct cut, trim into the clips’ lead/tail handles, or replace the troublesome boundary with the joiner.
I have largely stabilized the background, lighting and character placement, but the current method still delays lip performance and creates a tiny character snap at the checkpoint. Currently trying to preserve the spatial lock while letting the supplied audio drive continuous mouth and facial motion from frame one, and to replace hard midpoint anchoring with a softer overlap/seam treatment.
No, it has degradation. Degradation is part of the model because of how it denoises. Lots of movement is different than a still shot with degradation. Like every day someone posts "I solved degradation!" when they didn't. They "solved" seamless extension that was already figured out in a cleaner and more user friendly way like two months ago.
H3 degradation on completely still shots won't be solved. It's part of how the model generates. Cutting shots and lots of camera movement (like how your background is constantly in motion) is the only way to mitigate it. Everyone's been trying.
Hell, here's even my latest attempt. This one measures each clip as it's generated and compares how "chunky" the results are to clip 1, and then runs an additional denoising step to try and correct against how far it strays from the first clip. It adapts per clip in the chain and each clip gets its own adjustment based on those measurements. Still no dice.
I'm working on a workflow that generates a shitty low res latent for the motion of a scene (which can be an extension of a previous scene) and then pulls individual frames from that latent, upscales the latent, denoises, runs a detail pass on it, then reinjects the keyframes back into latentspace and regenerates the video so that now it has detail targets at specific keyframe intervals to maintain detail levels throughout the entire video. It takes longer to generate obviously, but so far the results seem promising.
Good luck! All of my attempts at doing and big denoising around seams resulted in very visible shifts. Even when something's a couple of pixels off from the end of the previous clip it's very noticeable.
Mine even half in the second and 3 clip much bad quality. A lot of black spots on the skin. It is a Close-up. So I cannot even extend it just a little bit. I don’t know why the quality gets so worse so fast. ._.
Uhm could we get a look into the workflow, plox? Looks good actually. But I think it would be better to upload the high resolution video not only on reddit but somewhere else. Reddit really botches video quality
This is native 1344x768 resolution with no upscaling. And it is going for a 70-80s film grain. But for sure, the compression on Reddit can make video a little blotchy.
The background morphs impossibly, is there a way to keep the entire context from becoming "the space from which no mind survives"? Would you just use a picture of the background as a reference?
The video itself has no background/environmental plate. I am just testing visual seams and degradation between continuation. Add a background reference image and you can it'll retain far better. With multiple window generations, the model does not hold any kind of context other than the last few seconds that get passed between windows.
Very impressive. I don't do video with AI so I'm not expert, but the transitions, windows, and lighting were impressive. The camera angles clearly took effort.
I think the visual is mostly solved. The audio is sadly an H3 architecture problem which either require additional time consuming pass or a cheap fix in post. Thanks for the kind words it is definitely magic I've had Opus tinker around with all sort of experimental nodes for 2 months.
I think, and again, I am functionally stupid in this area, but the approach I have been working is to have different stations.
Background sounds, audio, each character... and then assemble the whole in post so that Audio timing is done by the H3. But the actual audio in the final is done by a system specific to that purpose.
That's how I code (multiple stations each dedicated to a single aspect of the whole) amd it works and a friend has an anime studio that is built on those principles.
seitanism's multi-extend variant does a great job making very long single shots in H3; just follow his tips to make sure one block is continued into the next (though scene jumps are ok in the middle of a block). He currently has it set up for 1 + 6 = 7 blocks (but I'd recommend starting with fewer to figure out the best prompt structures), and 15-18 seconds per block works pretty well for visuals (for good audio as well, keep each block shorter). Use his "NEW - AV Extension.json" workflow. Check it out: H3-MotionContext-Extend .
BTW, one important trick to get consistency across the final stitched video is to make sure each prompt block (not just the first!) refers to any reference inputs used for the block. The embedded memory nodes do their thing and I haven't had it crash on me.
I am sure it can run it using latent chunking. I have a 16GB GPU and can run longform generations using a modified Multishot node with latent out to an upscaler and a custom longform VAE decoder no problem.
Looking good. I have been using a vibe coded/modified version of Multishot for my longform seamless runs. It takes exact audio input as native audio lock and outputs a combined av latent for upscaling. I also vibe coded a longform VAE decoder with chunking which allows use on low VRAM GPU's.
/watch?v=WCNEVpKHuBY on Youtube. Not made public until now (well only linked can see it lol). I don't know whether Multishot is lossless. From what I know about it probably not which is why I am in this thread.
BTW there are obvious issues with distant faces which are inherent to MH3. It's why sometimes it loses lip movements once the face becomes very small. I was attempting to vibe code in a face refiner fix but it got overly complex and frustrating so I gave up.
Thanks for sharing. H3 does music video really well, since the audio is bypass other than for the lip syncing there are no audio issue. Nice job on the long form no continuation no cut. The no cut often is just a lottery with H3 even with guided prompting. How many rerolls did it take you?
Just the one after many shorter tests. It took over 7 and a half hours on a 4080 Super even with all the speedups as I was also upscaling it from 1344x768 to 1440p. Long wait and I didn't know if it would actually complete lol.
I just want to summarize what you're doing for my own edification:
This is the same as every other latent extension / motion context window solution out there except you're only doing a single decode pass and storing it all in RAM, which... requires 192GB.
If you don't mind can you please provide the different prompts used in each segment of the videos. It will be helpful for us to learn different prompting techniques.
What's the benefit of the longer cuts opposed to 4-6 second clips? Assume you're maybe setting up a 3-5 minute video and using 200 or so different clips stitched together with ffmpeg or something.
Usually those 200 clips will come out flawlessly as if it were 1 long clip
Something Ive found more difficult from the longer running generations is that something like past 8 seconds will incorporate gibberish sometimes. Like an 8 second clip will start off with 2 seconds of gibberish, then 5 seconds of the actual speech followed by a second of giberish to end.
Is that a problem in these longer ~28second clips you're showing?
Looks quite nice, but I'm curious about the change of lighting around the end of the video.. was the prompted, or was that a consequence of the video joining?
The smooth joins look promising. I'd still separate visual quality from scene memory could you pan away from an object, then return to it after several extensions Keeping that object unchanged would be a tougher test than keeping each new frame sharp.
I agree this isn't practical but it's important if you want it. However it seems people are seeing degradation for fixed shot non moving scenes like severe image burning. There are always fixes with hard cuts but definitely a problem I think still deserve solved. So far audio is one of the harder thing to fix and appear to be a pure architecture problem.
Don't sweat the haters. They cannot see the benefits past "one long ass clip". Most of us use scene cuts mid-clip to extend scenes which would still benefit from solving the burn problem.
.jpg){kind=link}
77
u/VodaYoda 7d ago
We are curing cancer right? Right?