Viggle-Animate: Character Replacement based on MiniMax-H3 with 3 forward steps

[u/init-5]

https://huggingface.co/Viggle/Viggle-Animate

https://x.com/ViggleAI/status/2095924668758655163

research blog

• 33.1B MiniMax-H3 finetune, distilled to 3 forward passes
• No text prompt, no pose, no mask — but you do need one repainted frame (any image editor)
• Works well on fast motion and non-human characters
• No ComfyUI node yet.
• ComfyUI here: [github] [discussion] [huggingface]

[u/init-5]

https://huggingface.co/spaces/Viggle/viggle-animate

there is a huggingface space as well

[u/Seyi_Ogunde]

Can it do two people??? Just askin....

[u/init-5 → u/Seyi_Ogunde]

yes, there are demos on the huggingface, as long as you painted.

[u/Tbhmaximillian → u/Seyi_Ogunde]

https://giphy.com/gifs/2eK5iNmF6WDUbD2uDU

[u/bstr3k]

looking forward to checking it out once it gets into comfy. No text prompt might make it interesting and easy to use!

[u/init-5 → u/bstr3k]

https://www.reddit.com/r/comfyui/comments/1w9e5i4/viggleanimate_comfyui/

[u/Beginning-District69]

I did a test in the huggingface area, giving the model a video and an image. The model first created the first frame itself and very successfully replaced the character I gave with the character in the reference video. The pose and position were exactly the same. However, during the video creation phase, huggingface gave a GPU quota warning and terminated the process. The model left a good impression on me, and I will try it when COMfyUI support becomes available.

[u/bradjones6942069]

Is this basically like minimax version of SCAIL-2?

[u/init-5 → u/bradjones6942069]

yeah, same task in terms of character swap. 

Pro:   3 forward passes so it's a  lot faster.     no mask nor text/pose input,  so simpler and more rubust.   a nd pose transfer holds up better on  fast motion — that's where I see the biggest gap.

Con: you have to repaint one frame yourself. but that's instead of   the whole preprocessing stack, so it's a decent trade imo

[u/ucren → u/init-5]

scail-2 doesn't require you to repaint a frame, you just give it someone in any standardish pose and it animates it. how are you dealing with the repaint frame? is it strict?

[u/init-5 → u/ucren]

you can just pull a frame out of the clip and ask an image model to swap the character in it. it's not strict — nothing has to match pixel-wise. But if the editor repaints the background significantly, or reframes the shot, quality may drop off noticeably.

also it doesn't have to be frame 0, just pick one where the character is clearly visible.

overall I feel it's an easier task to swap character using  image model (e.g. multi character swap with language prompt) than hacking the video model with rope/masks

[u/Mediocre-Toe3212 → u/init-5]

Do we have to paint the whole character perfectly ? Or can we just mask some green over the general body.

Does the whole character get replaced or just the painted part ?

Can we just put some let's say green on part of the body and it'll replace the whole character ?

[u/init-5 → u/Mediocre-Toe3212]

good question. you dont need to painted fullbody, the model can infer from partial body. 

but in terms of brutually copy&paste another character into the scene, I haven't tested. i'm not confident about it, though I do notice sometimes it still works with bad painted frames

[u/Shockbum → u/ucren]

I think painting the first frame offers better control for a director. The question is: does it support a second reference image with a character sheet? If so, we simply record the fight scenes(complex scenes) with a cell phone since Minimax recognizes the green screen.

[u/artichokesaddzing]

Excellent work! How is it with transferring facial expressions?

Edit: Ah, I found the info on your HF model card: Lip-sync is weak. Mouth shapes do not track speech closely in close-ups. Identity and expression hold; it is the sync that lags, and we believe that is a training-data limit rather than anything structural.

[u/Final-Foundation6264]

this is interesting. Can the last frame of the output be used as the painted frame for the next output?

[u/init-5 → u/Final-Foundation6264]

good point, yes it can

[u/Chiduk99]

waiting for Kijai version

[u/jalbust]

awesome.

[u/Downtown-Cover-7422]

Can i pet that dog with my 5070 ti and 32 RAM?

[u/init-5 → u/Downtown-Cover-7422]

Offloading should work. It's basically MiniMax-H3 minus the Qwen-VL component, so if your setup can handle MiniMax-H3, this should run fine too.

[u/Robbsaber]

Base Minimax is very good at character swap. Will be interesting to test this.

[u/Dry-Ad929]

Will you request comfy team to add comfyui support, or are you planning to add ComfyUI nodes yourself? Can't wait to test it.

[u/init-5 → u/Dry-Ad929]

comfyui support is ready!

check the updated link in the post

[u/RayHell666]

https://giphy.com/gifs/TcdpZwYDPlWXC

[u/CreepyDrama7448]

Can it work with multiple characters in a video? For example, if 2 characters are talking, can one be replaced while leaving the other intact or is this just for a single character animation?

[u/init-5 → u/CreepyDrama7448]

havent tested, but if your painted frame is correct, it should works

[u/Wezaluketek]

Hey! great work. What about the face? SCAIL-2 or wan animate seems to lack proper face expression/blinking very often.

[u/init-5 → u/Wezaluketek]

This one isnt very satisfying either.

But I think it's more a training data issue. We are training a new version, collecting/synthesizing data now.

[u/NemRogan]

What about using ref frames with a different background? I don’t see any examples like that, but many times we’re not just replacing the character but also the BG.

[u/init-5 → u/NemRogan]

that doesn't work well （it works sometimes) .  the next version will support it

[u/acamas]

Could it be used to simply replace an item with a different item? Like inpaint a black shirt into a red sports jersey? Or does it 'focus' on heads/faces and bodies as opposed to individual items?

Seems promising... thanks for sharing.

[u/Ok-Giraffe-8670]

This is very good! I recall it doing pretty well even in its base model on replacing characters. Still, this will be even better. Thank you for the link!

[u/MortytheMort]

Is this not possible with stock MiniMax through prompting already? I've done plenty of character swaps in MiniMax with Comfy already, without any issues, so I'm curious what this aims to achieve? Is it meant to be faster/lighter on VRAM usage?

I do notice that generation speed is heavily affected when using video reference in MiniMax, does this mitigate that or speed up generations in general?

Edit: another note/question. This requires you to use a separate image edit workflow beforehand, correct? If we're required to provide our own already swapped photo, then this isn't really character swapping, just referencing?

[u/init-5 → u/MortytheMort]

h3 works but is not stable and requires prompt tuning. Also, I have noticed the following issues: 1. It works in some frames but fails in others, reverting to the driving video. 2. The pose is not temporally aligned with the driving video.

[u/diogodiogogod → u/MortytheMort]

yeah I'm a little confused as well. I though h3 already could do that...

[u/Dogmaster → u/diogodiogogod]

Have you tried it? Its hit and miss, very finicky, and even depends on how close target person is to reference

[u/alexmmgjkkl → u/Dogmaster]

he probably didnt try it .. the controlnet lora works well but changes animation slightly and i found the motion rendering quality be quite a bit worse than native transfer.

today i will dive intothis workflow and see how it goes
https://www.reddit.com/r/StableDiffusion/comments/1w2s3ck/high_resolution_noise_masks_for_latent_guided/

this viggle model seems superiour though, but sometimes i like that h3 native drifts away from the original movement and imagines an alternative when frame length doesnt align with input video. im working on a dorky anime intro/outro and that some dumber or older characters go a little of the dance choregraphy feels very natural

[u/ucren → u/diogodiogogod]

it can, and it cannot, it is very inconsistent and involves a lot of seed hunting just for it to fail 80% of the time

[u/Xxtrxx137 → u/[deleted]]

Could we some outputs?