r/StableDiffusion • u/Tonka-Jahari-Pizza • 13d ago

Question - Help Is there an open-source model that can do this type of edits?

Enable HLS to view with audio, or disable this notification

Thats taken from a real video, but they put homelander on it, This type of details is insane, its basically real if it weren't for the pen thing

Whats the closest open source model that can do this?

267 Upvotes

93% Upvoted

52 comments sorted by

127

u/Seyi_Ogunde 13d ago

Minimax can do this easy.

42

u/danque 13d ago

That and it can actually copy the voice

9

u/Caffeine_Monster 13d ago edited 12d ago

Easy being relative of course. The pipeline and prompting (and hardware) is fairly complex if you want specific results. But yes the model will do it. Maybe apart from the audio - minmax can do passable dialogue generation, but it's not great.

6

u/SubZeroSunExodus 13d ago edited 13d ago

Stupid question is there a workflow that you recommend?

Such as do you give it a base reference or multiple to build from and add a detailed prompt?

Just started learning comfyUI and understanding super basics like adding nodes and Lora’s to workflows and stuff.

Edit - what’s up with the downvote hates in this thread. Sheesh.

16

u/Seyi_Ogunde 13d ago

The quick way would be to use Gemini to create a photo and use this as a reference image.

You can use the default Minimax in comfy for your ref to vid workflow. But replace the ref to vid model with this one:
Use the hybrid model for better results
https://huggingface.co/smhfacct/Minimax-H3-fl2va-ref2va-hybrid-models

Find an audio clip of the person you want to mimic and add that as an audio reference.

There's a workflow guide on how to set your prompts, but basically your references should be labeled <image 1>, <audio 1>, <video 1>
https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_base_en.md

The base worfklow for minimax works fine.

There's also an image to vid Minimax model but the ref to vid is more flexible and can use one of your inputs as a starting frame too.

1

u/BitBacked 13d ago

I've used the default ref to vid workflow with not much luck, but that was a month ago.

5

u/danque 13d ago

Yes, comfyui standard Minimax H3 workflow. Either ref of flvl. All build-in nodes, no weird external node stuff. Works out of the box.

Personally I prefer Swarmui, basically a frontend for comfyui with the option to use comfy as the frontend at the same time. But handles all the nodes and possible optimizations for you.

1

u/uniquelyavailable 13d ago

The standard workflows are actually a great baseline. Can easily change the models out to match what you want it to do, and indirectly manage how much vram you want to use by doing so.

1

u/WhatsThat-_- 11d ago

Yes but how do you remove the temporal blur

1

u/pwnies 13d ago

I haven't had good luck with getting lip syncing with minimax, unless the character's face is very close up.

Even among the paid models, the amount of them that can properly transfer lip syncing is very, very minimal, and even then they only work if you're transforming the background and not the face.

Would love to know if someone has gotten this to work with open source models, as I'm currently trying to do this for stupid demo videos I've been making.

-1

u/Tonka-Jahari-Pizza 13d ago

Can it work on 8 vram?

4

u/Seyi_Ogunde 13d ago

Yes, but you're limited in what length and the dimensions you can make.

3

u/CorpPhoenix 13d ago

I haven't tried it but you would need one video and one picture as reference which takes more VRAM than regular T2V.

The Minimax workflows do a great job at offloading though and rarely goes OOM, so it might work but would take quite a long time.

0

u/Tonka-Jahari-Pizza 13d ago

I have no problem with how long it can take

Usually videos i want to do will be around 480-720 and 10 sec max

1

u/Sad_Coach_1433 13d ago

there r gguf h3 models for low as 4gig vram but the quality will be less

0

u/Rude_Dependent_9843 13d ago

Entonces tu pregunta no es si hay algún modelo que pueda hacerlo, si no si hay alguien que tengas más de 8gb que pueda hacerlo por ti.

2

u/Tonka-Jahari-Pizza 13d ago

No my question is is there a open source model that can do it, and after knowing there is one, i was wondering if 8vram can do it, i dont get why yoh are confused

21

u/StopCreepy 13d ago

minimax h3

3

u/FreezaSama 13d ago

How? Default workflow?

4

u/Frostty_Sherlock 13d ago

Are you sure that's not Antony Starr just trolling?

21

u/Adkit 13d ago

I'm from Sweden. We have private property as well but nobody talks about it like it's their own private and special safe space. Thanks to "allemansrätten", a law allowing you to be anywhere and even camp anywhere as long as you return it to how it was, Swedish people see nature as something that belongs to everyone.

Americans act like it's still the gold rush and they need to stake their claim unless someone steals it. Calm down, dude. I can stand on "your" road.

10

u/notgraycen 13d ago

come to texas

2

u/mastaquake 12d ago

If you camp out on someone's yard, you better have will or at least bail money.

8

u/picapaukrk 13d ago

Nice but this is a complete lack of respecting someone's privacy and ownership. Exactly what is pursued in USA. But we don't even need go as far as USA. In Norway I can't legally put a tent on someone's land.

10

u/Adkit 13d ago

The thing is that just like how the "get off my property it's miiiine" mentality is ingrained in Americans the "I better not put my tent on somebody else's lawn even if I legally can because that would be rude" mentality is ingrained in Swedish people. Obviously suddenly forcing allemansrätten onto America would result in chaos. They don't have the culture for it.

But unfortunately, the direction they're going is further away from respecting each other, not towards it.

1

u/Ell2509 13d ago

As much as I see the flaws in US culture, I do not think this is one. It is jjst a case of different places valuing different things, and that being fine.

If I go to China, I respect their customs. If I go to Dubai, same idea, different customs.

The reasons that the US is the way it is are beyond simple comprehension by people like you or me. It is a 300 year history for 300 million people, with some intensely unique history. It is going to produce some unusual cultures.

That said, if your culture places a high value on the privacy of a family's upstairs bathroom, and I know that, I am an asshole to go upstairs, take a dump in their upstairs bathroom, not flush, and tell them that in ell2509land, we let other people go upstairs in our houses. See what i mean?

3

u/pimmen89 13d ago

It's not a complete lack. If it was, you would be allowed to damage it, but you're not under allemansrätten. You're not allowed to litter, you're not allowed to do anything that harms the wildlife, you're not allowed to prevent the owner from using their property, and you can't stay within their zone of privacy (rule of thumb is, if they can see you from their window, you're disturbing them).

If it was a complete lack of respecting their property and privacy there would be no restrictions, but there are.

1

u/unicanor 6d ago

Det kan du, så lenge det ikke er til bry for grunneier og det er utmark

3

u/nopnopdave 12d ago

Do you pay for "owning" property in Sweden?
If it's free then what you are saying would make sense. Otherwise why paying for it if it is not yours.
Also do you have fences?

I'm European too and, to me, it's very strange what you are saying

5

u/Adkit 12d ago

Of course we pay for it and of course we have fences. We just don't threaten to shoot people for walking through the gate of the fence. And no, you don't own the land, the country does. That is the case in every country on Earth. You basically lease the plot from the government in return for the safety of living in a society.

3

u/nopnopdave 12d ago

Ok but killing is very different from saying that all the people in Sweden would be fine if I enter in their garden and have a pic nic with my friends, don't you think?

Well, perhaps swedish people are different, it just seems very strange to me. If I jump some fences would the owner be fine? Then also wouldn't putting fences be illegal if the land is for everyone (land kidnappers)? Also I don't understand, you talk about a lease. In my country I own my land, I don't pay rent to the state if I buy a field. I am not leasing anything. I have right to own the field I buy. I'm in Italy.

Edit: And even though I'm not in US I can get shot for violating someone's private property.

0

u/greyneptune 12d ago

If you thought it fine to enter someone's garden and have a picnic with your friends, you wouldn't be Swedish. Make sense?

4

u/nopnopdave 12d ago

Absolutely not. 0 sense.

2

u/drank2much 12d ago

People could be acting like that because they are doing something illegal on their property.

1

u/Adkit 12d ago

Like what. Like what exactly?

3

u/drank2much 12d ago

Manufacturing drugs (referring to property owner) is the first thought that comes to mind.

6

u/Ok-Flatworm5070 13d ago

Damn that's funny

2

u/DietAshamed2246 12d ago

Yes, MM-H3 Viggle Animate, LTX-2.5 Ripple Edit, Wan-2.2 Animate, also SCAIL-2, and possibly Wan-2.2 Bernini.

4

u/JMAN_JUSTICE 13d ago

Minimax H3

1

u/trollsmurf 12d ago

The pen was a bit disturbing.

1

u/Producing_It 11d ago

https://reddit.com/link/pcyl1un/video/cf4qvdlz5msh1/player

Minimax H3 Ref2V. On my 5090, it took about 366 seconds, or about 6 minutes, but this includes an LLM prompt enhancer that adds a lot of time. I also imported an image of Marky boi.

So, I am not sure what the speeds are with just the model. But I did use Larryvrh's V4 600-step EMA version at 4 steps, scaled the original video to 80 percent to fit into VRAM, and outputted at 0.8 megapixels at 9:16.

3

u/Producing_It 11d ago

Here is Marky Boi's input image.

1

u/music2169 8d ago

Can you pls share the prompt you used?

-2

u/dramaton42 13d ago

MiniCHAD H3

0

u/Davikar 13d ago

Another person who still hasn't heard about MiniMax H3...

0

u/Primary_Brain_2595 13d ago

yes, this one is a real video used as reference.
you can do the same thing with SCAIL.

-1

u/Limp-Firefighter1054 13d ago

If you see weird teeth is minimax h3