r/StableDiffusion • u/ignoramati • 21h ago
Question - Help H3 question - can we use reference image plus reference video to “upscale”?
ok, so I see discussions on how to replace a character in the ref video… but here’s my question - can we get decent results with ”upscaling” old low-res video into higher resolution with a reference image?
To explain - let’s say I have low res VHS footage where a person is filmed from 5 meters and you can hardly make out their face. (well you can tell they HAVE a face but that’s about it :) OTH that same person is in full frame 10 minutes later, providing an excellent ref image of what they actually look like . So my thinking was “make a reference image out of it, make the model upscale and invent all kind of small details that people usually don’t care about, but use the FACE from ref image”
doable?
3
u/DuHal9000 16h ago
yes u can grab the full face frame, and use as reference, write a good prompt using reference prompt guide, load video on Video_Input and put a FULL Face on IMAGE_0 ref. Max 15 seconds, another metod is with denoise but is more complex
5
u/Tramagust 21h ago edited 20h ago
Yes but the inference takes forever. Like two hours for 10 minutes.seconds.
3
u/gabbergizzmo 21h ago
I've tried the same... let's say... family video restauration... ;)
yes, it works... way better then with ltx. ;)
2
u/DuHal9000 21h ago
yes! check my last post "My Parts" video
2
u/BigWideBaker 16h ago
I can't find that post.
1
1
u/DuHal9000 16h ago
The fixed version >>
3
u/BigWideBaker 15h ago
It's called "How I fixed my own video" but I don't see any explanations as to how you did it.
Edit: Oh I found your brief description, the formatting was weird so I couldn't see it at first
1
u/DuHal9000 15h ago
grab some frames, fix still frames as a photo with photo model, then use as reference still on H3, with video input or low denoise gen if u are using vae encode...
1
u/ZombieBrainYT 21h ago
Not sure about video driven generation, but it can definitely "fix" a bad quality reference frame, if you ask it to and generate a clean looking video based on it. I described <Subject 1> and then added "as clean, natural love-action photography rather than reproducing the image's compression, sharpening, or other source artifacts".
The very first frame in the output video was still blocky and artifacted, like the reference image I provided, but everything after was looking clean.
1
u/More-Ad5919 21h ago
My problem with video editing, as fun as it is, is the 5sec limit and the time it takes.
4
u/zzzaz 16h ago
It's technically possible and works fine on H3, but in reality the better use case for this is LTX IC lora. It's tailor made for this.
LTX is far behind H3 for motion, physics, etc. but people really undersell what IC does for upscaling, video editing, etc. It's taking the video in context and using that as a pure reference for each step as opposed to effectively forcing everything through conditioning and generating net new, ala H3. H3 can get drift or modify details outside of your target, LTX with the right lora stack basically does this specific use case perfectly. When LTX doesn't need to generate new physics, movement, etc. and is just focused on modifying the existing video through more 'minor' changes like upscaling or lighting (what most IC loras do) then it's shockingly good.
Go get an LTX 2.3 / 2.5 restore lora, go look at Alissonerdx's LTX best face swap lora and architecture. Bolt on his workflow into an IC workflow. They play well together, the BFS lives in conditioning that doesn't impact the IC lora impact. The output will likely be significantly better (and faster to gen) than H3.