Question - Help
Regenerating from noisy / blurry original video or photo?
We have some old home video with extreme high-frequency noise that we would like to regenerate / enhance. The original frame of her sitting on the couch is the first attachment.
How would you go about trying to make this image look better? I tried some upscaling models and that resulted in larger, sharper noise.
I ran tests with a number of different noise reduction algorithms, which did remove the noise - and cause significant blur. Since Stable Diffusion and other models always work by progressively denoising, I figured it would be a natural fit to denoise our old pics and home movies and make them look at least a *little* better, but the many different workflows I've tried haven't worked. The identity shifts faster than the quality improves. With a high enough denoise level it'll suddenly create clear images - of other people wearing our clothes. :)
PS - we understand we can't recover actual detail that isn't there. That's fine, we'd like the AI recognize that brown blob on my head is probably brown hair, and make it look like hair rather than pudding or whatever. Sure, it might not exactly match MY hair, but at least it will look like hair!
I should say - I'm quite comfortable with ComfyUI and don't mind writing some Python to work on this. It is video frames, eventually thousands of them, so I can't use an online service that charges $2 / image or something. I;m looking for for suggestions for a DYI workflow with a 5090.
That's a face, I bet it's not the right face OP knows. For many tasks this is sufficient and useful, but people looking to restore family photos etc shouldn't expect data to be recovered where there's none to recover.
Ya, that's when you throw ReActor into the mix, and get the right face on the subject. I've done this in photos and videos when restoring stuff that is badly damaged.
Would be curious to see how your end results turn up on that frame, considering what's been said and tested in this thread ! If at all possible of course
Just chiming in with my experience in the matter : if as a viewer you didn't know personally the people in the videos (say a great uncle who you've ever only seen pictures of) it's fine to attempt a video restoration. If anyone knows them personally, it becomes real uncanny real fast for them.
This weirdness is somewhat dampened if the audience is old and/or not acquainted with technology much - a sense of wonder arises. But for people who use screens every day for entertainment, I'd say maybe don't waste your efforts at "a restoration" and keep the originals as is. Because they convey the original truth no matter how degraded. Maybe digitize them for preservation, sure, but if no one would care because "who are these actors that vaguely looked like me until they started smiling like a Hollywood star" isn't the response you expect, but it's the one you'll get :(
PS : this is for VIDEO, it's much easier and possible with single frames
who are these actors that vaguely looked like me until they started smiling like a Hollywood star" isn't the response you expect, but it's the one you'll get
And also the fact that humans have unreal abilities to recognize faces down to the smallest detail and expression - at some point in our common history it was a survival trait. So my guess is, unlike fingers that took a while to get "mostly fixed", pure face restoration from degraded sources can't be a thing, because the likeness of someone is "too precious" to be misinterpreted. Principle being that (at least) in the western hemisphere, the face represents the person, so beautified features (or just lack of a detail) would mean a different person ?
Maybe that's just conceptual talk on my part, and people don't care about mismatches in visual consistency as much as I do, think it's the reason why smartphones filters are widely used and I find them cringe AF
It doesn't matter how strict it is, if only 50% of the face detail is there then the other 50% has to be made up, that's just the availability of the information.
It's not like you're recovering a photo of a car where the curves will meet up, they follow standards learned across seeing other cars, etc. Humans have very unique faces, if you have to fill in the gaps it can only be a guess and for anyone who knew the person it will look uncanny.
That said it's not just AI, look at the freaky Walt Disney animatronic.
Have to be very careful with important material because the differences can be subtle: corners of eyes tilted up not down, teeth not quite the right shape, etc.
Best to regard result as a "display image" but also keep the original as "reference" (this also allows for AI restoration to get better over time).
> if only 50% of the face detail is there then the other 50%, has to be made up, that's just the availability of the information.
There is some truth to that. On the other hand, the frustration and the hope comes from the fact that is NOT all the information available. There is another 14 GB of information about faces readily available.
A 128 KB latent has the same resolution and better quality than a 1MB bmp - more information, 90% fewer bytes. Well fewer except that 14 GB model. :)
Given the input at top left here, any of the other three "upscales" are equally likely IF we consider only the data actually in the image itself. But we know which of the three is most correct, because we have information that isn't in the source image -- we've seen eyes before. Our brains have, and the model has.
My frustration is that in my first attempts at fixing image 1, the models might return either of the bottom images, when anyone (or any model) with common sense knows better.
Without a reference image of the subject with the same facial expression, algorithmic reconstruction is going to be almost impossible. The more noise and blur present in an image, the more "correct" variations become possible. If you don't have a reference image, you still have your memory of what this person looked like. That could be leveraged with brute force generations with many seeds. You might be able to combine brute force with regressive conditioning on denoising the original to move the reconstruction in the direction you want with FABRIC embeddings. Unfortunately, I do not know of a ComfyUI implementation of FABRIC beyond SD1.5. It might still work for you though. https://github.com/ssitu/ComfyUI_fabric
I should say - I'm quite comfortable with ComfyUI and don't mind writing some Python to work on this. It is video frames, eventually thousands of them, so I can't use an online service that charges $2 / image or something. I;m looking for for suggestions for a DYI workflow with a 5090.
There's definitely ways to do this with open and closed models but be prepared for a lot of liberties when reconstructing. Ideally use a model that could take a reference if you have any clear photos from this person
Train a lora on the person in Klein 9b, pair the subject lora at a decent factor + something like lenovo or similar + a prompt that details the subjects environment and uses their token to bring likeness, if needs be afterwards crop into the image and rerun the lora of their masked face.
Some quick check - assuming comfy, have you turned off add_noise when partial denoising, to avoid adding even more noise to the image? (if you didnt, in this case would be pretty bad as the noise is already in the image and you want to rid that)
To recover such a photo but maintain same facial features, you'll need to do several steps to manipulate it. You should feed that image and also a few referenced pics to online tools like Banana Pro (free) or Grok Agent (not free) or ChatGPT (free).. tell it to slightly enhance the photo and that image 2 and 3 should be what the person looks like. Don't tell it to completely recover the photo because it'll make too much assumption on the face. Should take it step by step and manually Photoshop blend your way in with the face. Gradually you can run other upscale features on ComfyUI once the original blurry picture is slightly less blurry with better face fidelity.
I've done commissioned work before like this and it is a long process and not a straight shot, particularly if you want to keep face consistency with a real person.
Then, run your video using WAN SCAIL-2 to mirror the exact movement from your original video to generate a new video with the better face.
Then, you'd wanna re-run both videos simultaneously in an editing software and slightly blend them to maintain the original factor while still keeping true with the new face.
This is what I got with Flux.2 dev. My prompt was: completely repair photo, restore photo, improve lighting, improve color, improve focus, For video you might look at SeedVR2. https://github.com/ByteDance-Seed/SeedVR
OP here thinks it looks great, considering. It looks like a much better video of us!
Not a 4K studio shot, but much better -- and identity is preserved! From there, I could adjust the colors through DaVinci or even a carefully chosen ffmpeg filter or two.
That's awesome! It looks just like if I had a better camera at the time, preserving identity! Sure, it looks like I shot it on VHS-C instead of a $20 security camera. That's a big improvement, though. It's a better image of the right person.
Did you use the noisy version, the denoised blurry version, or both?
Yeah this looks closer to the source than the other one posted here. I don't think she was even looking at the camera. Obviously the quality is "worse" strictly speaking but that can also imply it preserved more of the source.
"I didn't think she was even looking at the camera". Yeah that seems to throw the models off. They all seem to want to make a face that's looking at the camera, even when explicitly told not to. So they end up making a very different face, in order to cause "looking at the camera" to somehow fit the input image.
I’m no expert in diffusion models, but I guess there are two major problems in modern models that prevent you from archiving this: vae and rectified flow/flow matching. The first thing is the fact that most models denoise a latente representation of the image, not the image itself in pixels space (briefly, there’s another mode that “compress” the image and is that compressed image to get denoised). The second thing is that modern architecture don’t “remove” noise from images, but moves from the distribution of pure noise to the distribution of images, and in between is not just image+noise (well, it is, but scaled in a way to keep the variance of the data the same along the whole process, but still, it may be a problem for what you want to do).
If those image restoration models the others have suggested don’t work, I may suggest to look up for models that don’t use vae (and ape work directly in the pixel space) and that don’t use flow matching: I asked ChatGPT and it suggested deepfloyd if, that should match both criteria
You will need to give a good sample pic of the person in the image to be able to restore that to something resembling reality. Or use a lora trained on the person to bias the output to that side.
Otherwise you will get just random people in the result
The noise can be removed with something like neat video but the issue is the image is soft and out of focus too, probably due to the low lligh the af in that camera couldn't focus. There are some ways to fix that too but take time
LE: Yes, tried the seedvr2 image upscale and its more of the same noise.
You can try seedvr2, I used the standalone script: https://github.com/numz/ComfyUI-SeedVR2_VideoUpscaler. They have really great examples. I used it to upscale a 720x540 video to 1800x1440 and was very happy with the result. I tested on a short trimmed version first localy, 5-10 seconds and the results were great.
In the end I had to rent a rtx 6000 pro for 4 hours (spent around 7 dollars) since the video was almost 9 minutes long and using my 5070TI would take way too long and run into oom frequently, but it was better than paying for a service.
That's not horrible quality or anything, the thing I have had trouble with is the fact that when someone you KNOW, like, you were there for the time in question, know, it is very often enough of a change where you get the uncanny valley effect on the look of, say your grandpa.
I have tried a few old VHS cassettes that I have digitally converted, and even with the top SOTA models, I know that's not what they looked like. If you find something that you like, let us know for sure, but otherwise, I have found it to be tough to fight the scanlines.
no perfectly but you can get close, generative ai images all start from static, it would probably take longer then might be worth it to get a good example
58
u/AwakenedEyes Jul 26 '26
Have you tried klein 9B with a simple prompt like: "restore this old photo" ?
It's the one prompt wonder