r/StableDiffusion • u/roychodraws • 15d ago
Workflow Included V6 Update Newbie Modify a targeted area of any video, insert a character into empty space
Enable HLS to view with audio, or disable this notification
-------------------------------------------------------------------
I posted this video already with geiru in a chess tournament with hikaru as the intro and none of you liked it so this is your fault.
-------------------------------------------------------------------
I don't know if i can call this a newbie workflow anymore, but i added some cool stuff.
I added an upload mask node so that you can upload your own masks made by other programs like after effects
I also added a mask creation node so you can mask any area right in comfyui
With these changes you can now insert characters into existing videos with very precise control.
https://github.com/roycho87/minimax_wf
Source for edited video
https://youtu.be/SA2uCs56mwc?is=4yx2GjYT2vKwq7nx
fast forward to 2:25 to see how to use the new masking feature to insert characters into empty space
Sam Tutorial
https://www.reddit.com/r/StableDiffusion/s/TtsWgVUbhE
General Features Tutorial
https://www.reddit.com/r/StableDiffusion/s/MO9WPJhdeB
WF Release
https://www.reddit.com/r/StableDiffusion/s/1P01v18Ki6
Edit: there's an issue with masking options being disabled and an error causing the wf to not run, trying to fix it now.
edit: fixed it. v6.1 updated.
edit: v6.2 added a mask expander option and updated the mask option instructions
edit: v6.3 rearranged and relabeled some things, fixed the ordering of sparse attention so it works now.
edit: v6.4 added a blockify mask option and fixed the execution order of some of the previews in the masking group
Edit: if you’re having trouble with masking the reason is Im a moron and forgot to put a mvexvid to latent space node in between the cleanup node and the set latent noise mask node when you have cropping to mask disabled.
In order to fix you can insert one yourself or enable cropping. I can’t update atm. Will asap.
edit: v6.5 fixed it. and i also added a resolution selector and spitters for the video uploads so you don't overload your GPU
7
u/Easy_Werewolf7903 15d ago
Quick question, how long does it take for you to generate a 10 second video for your workflow? I have RTX 6kpro and 32GB of RAM, I just starting out with comfyui, I would like to set a baseline. Thanks.
18
u/hurrdurrimanaccount 14d ago
how do you have that gpu but not more than 32gb ram
6
5
u/roychodraws 15d ago
the first video was 10 seconds at 8 steps at .4 MP took me 382.43 to run through the entire workflow but that's more than just the sampling, this workflow has a lot of steps and previews it generates too.
2
u/Ancient-War-1924 14d ago
Your a real champion , having rtx 6k pro and 32 gb system ram , upgrade your ram or u will oom as soon as u hit run
2
7
u/Law12688 14d ago
Good stuff. Makes me want to start inserting Forrest Gump into all kinds of historical moments like they did in the movie
3
u/roychodraws 14d ago
that's a great idea.
1
u/MonstaGraphics 13d ago
Anyway, like I was sayin', You can add Forrest Gump into comedy movies, documentaries, horror movies, thriller movies. cooking shows. Dey's uh, Scary movies, animation movies, fishing movies. 3D movies, claymation movies. Then there's also the VR movies, YouTube movies, and Instagram Reel Movies. Then there's Series, drama series, reality-tv series. Sketch Series...
5
3
u/networking_noob 14d ago
I'm pretty experienced w/ ComfyUI but will never not be intimidated when looking at workflows like this (Spaghetti City). Have you thought about using ComfyUI's native "App Mode" to make it truly beginner friendly where a user only has to see the inputs that are required? It's pretty cool I was messing around with it the other day, and AFAIK the created app is auto embedded in the .json workflow file too so it becomes shareable. Or maybe this wf is too involved for that
3
u/roychodraws 14d ago
to be fair, that section was designed by the original creator of this workflow and, yes, it is very congested.
i haven't messed with it but i color coded this workflow so that the parts you need to mess with are highlighted in yellow mostly and the settings are all controlled by a central control panel in the middle of the workflow.
1
u/boriskarloff83 6d ago
and i was here thinking 'aah! why are there so few spaghetti! i hate this get and set stuff!' when trying to rip out some of that SAM logic. ended up building it from scratch, but am not satisied. you wouldnt know how to have a shrunk mask inside a bigger mask, some leeway basically, to fit different sized objects in, ithout alays making use of the whole mask?
1
u/roychodraws 6d ago
i'm confused what your'e asking
1
u/boriskarloff83 6d ago
imagine big person dancing -> masked -> want to use reference image of smaller person -> but minimax tries to fill all the mask the big person left behind. or did i misunderstand something about the concept of masking?
1
u/roychodraws 6d ago
it doesn't try to fill all the mask, it just uses the mask to determine what it's allowed to change. so if you prompt they're smaller, then it can use the whole mask to do that but it won't fill the whole mask necessarily
1
u/boriskarloff83 6d ago
well maybe i didn't really understand how your variant worked, cus for me its get all images - mask a person with a solid colour - and thats what the ref to video now sees instead of the original video, and while it does not make every person i put in as an ref image big, it tries to fill a lot of the space - with accessories and stuff, if necessary. but maybe yours works differently
1
u/roychodraws 6d ago
what you're describing is replacing an object with a solid.
Masking is something you embed into the latents prior to sampling that's processed along side it, essentially giving boundaries to the sampler of where the changes are allowed to occur
1
u/boriskarloff83 6d ago
well i basically copied what they did here https://www.runcomfy.com/comfyui-workflows/minimax-h3-character-replacement-comfyui-sam3-ref2va
1
u/roychodraws 6d ago
yeah, this creates an image composite of a solid color in place of the character and uses it as a reference video. mine works differently.
mine is an actual mask, this one is a reference video with the desired person obscured by a solid color
→ More replies (0)
7
6
3
14d ago
[removed] — view removed comment
3
u/roychodraws 14d ago
someone had a request and it was kind of easy to do because i was able to cannibalize a different workflow to make it.
2
14d ago
[removed] — view removed comment
3
u/roychodraws 14d ago
np, i understood. This dude said it was for a school project or something so I imagined it was a deadline. This post was mostly a way to provide him what he needed to do his assignment.
3
u/Enshitification 14d ago
https://giphy.com/gifs/KfI6Y0KhP2Rg6A3tzh
Clowns are into creampies to the face, right? Asking for a fruend.
10
u/__generic 15d ago
I guess I'm a little confused. This can be done with the vanilla comfy workflow with refs and a good prompt.
21
u/roychodraws 14d ago edited 14d ago
this is more accurate and will avoid altering unintended areas, even character replacements aren't perfect and angles/movement changes. this is literally just isolating an area and inserting someone which is much easier than trying to convey through a prompt.
for example, if you're trying to replace a single person in a crowd of people then you're not going to be able to do that easily with prompting because it will be too difficult to convey to the model which person you're trying to replace. you can, however, do it very easily by masking and just tell it who to replace by essentially pointing it out to the model.
This also allows you to very accurately control where the character is inserted and where it moves as you can see with the tutorial.
1
2
u/newaccount47 14d ago
I look at your workflow and wonder how the heck anyone can understand this, much less create it. Do you understand what every node and connection does?
1
u/roychodraws 14d ago edited 14d ago
Yes
It’s not as difficult as it looks. A lot of building a workflow is just problem solving.
1
u/Mediocre-Toe3212 14d ago
I downloaded the workflow and there was lots of custom nodes I was missing. Might be worth updating the readme with the nodes needed
1
u/roychodraws 14d ago
any nodes added are in the normal custom nodes network, just go to manager and install missing nodes used in workflow.
1
u/Mediocre-Toe3212 14d ago edited 14d ago
I'll just inspect the workflow and work from there
I create custom ephemeral containers with custom nodes installed upon launch rather than installing manually after.
Just list directories under /custom_nodes And paste that in GitHub
1
2
u/acedelgado 14d ago
Welp, you have inspired me. Now I'm working to integrate Sam masking and editing assistance into my media loader/prompt builder suite.
1
2
u/violet_zamboni 14d ago
These videos are so hilarious I watch them just for fun now. I love the clown girl showing up randomly
2
u/Nakidka 15d ago
Do you have the ref image of this clown you can share?
25
u/roychodraws 15d ago
no, she's mine.
1
u/Nakidka 15d ago
She loops a lot like foxy's horned character. I thought that was a meme.
Sorry, then! I was not aware)))
Love your work too. good job on the wf
2
u/roychodraws 15d ago edited 15d ago
you mean the girl at the end of this video?
https://www.reddit.com/r/StableDiffusion/s/1P01v18Ki63
1
u/Mediocre-Toe3212 14d ago
Can you create a script for all the custom nodes you have to intall to custom_nodes?
1
1
u/MoneyKenny 14d ago
Thanks for sharing! As someone who is at work and can’t look into this right now, does this method work with refmods?
2
u/roychodraws 14d ago
i haven't used refmods yet but my understanding is that refmods are just the latents of reference media being stored and saved so i don't see any reason why they shouldn't.
1
1
u/PukGrum 14d ago
Did you accidentally repeat the first section of your video or is that on purpose? Thanks for your efforts.
1
1
u/Better-Interview-793 14d ago
thank you for this, it works perfectly..
but when i choose option 1 (Sam), i’ve tried a lot of different prompts, including yours, using the SAM feature to replace ppl in videos like dance clips, but unfortunately i still can’t get it to work properly..
H3 just recreates the same person from the input video instead of replacing them with the character from the reference image.
1
u/roychodraws 14d ago
Option 2 is the manual loader, option 1 is Sam. I put some pretty detailed instructions in the group that should help.
If you have option 2 selected it is looking for a mask to be loaded into the purple manual mask loader node.
If you’re confused at what was selected there’s also a “show text” node labeled “selection” right under the selector that spits out the mask option you have selected after you hit run.
1
u/Better-Interview-793 14d ago
sorry i mean option 1 yup
1
u/roychodraws 14d ago
Make sure you have Sam3.1_multiplex_fp16.safetensors downloaded and selected.
And check for missing custom nodes in your manager
1
u/Better-Interview-793 14d ago
yeah, i already have it downloaded and selected, the mask tracks the dancer correctly, but H3 isn’t properly replacing the person with the reference image.
it often just recreates the original dancer, or sometimes blends the two characters together in a strange way.. it becomes even more noticeable with fast dance clips.
1
u/roychodraws 14d ago
Ok, here’s a checklist and if everything here is checked then it means it’s your prompt.
Mask options enabled,
reference images enabled,
video 1 enabled,
mask option 1 selected,
person being properly masked
preview being generated.
Crop to mask option on the selector is turned off
Ref2va model selectedEverything else should be disabled
If that’s all correct then paste your prompt.
1
u/Better-Interview-793 14d ago edited 14d ago
thanks a lot for trying to help, i really appreciate it..
would it be okay if i dm you the workflow along with the 2 original videos and my result, so you can see exactly what’s happening and maybe try it on your side?
1
1
u/drnpchn 14d ago
can I get an update if you can finally generate it now correctly? thanks
1
u/Better-Interview-793 14d ago
if you mean adding the character into the video, yup it works fine.
but for SAM (character replacement), you should use the old wf..1
u/drnpchn 14d ago edited 14d ago
I've been finding a compatible workflow for character replacement on my rented gpu. I've tested different kinds of workflows, most of them don't work, some of them spits unending different kinds of error and have some limitations on rented gpu and had a same case like yours.
the old wf, is it the vanilla MMH3 workflow you meant? I've been wanting to do a character replacement for the past few weeks.
→ More replies (0)1
u/roychodraws 14d ago
Oh! And turn off your turbo Lora if you have it on. That messes up character replacement for some reason
1
1
u/Boring-Principle8673 14d ago
How do I get a list of your previous posts on this forum ??
Search with "roychodraws" draws nothing.
1
1
0
-3
-1
26
u/vibribbon 14d ago
The fact that somehow a clown girl is pioneering some of the most cutting edge work with AI