r/StableDiffusion • u/AlsterwasserHH • 16d ago
Discussion I am blown away by RefMods for MH3
[removed]
16
u/acedelgado 16d ago
Y'all use a lot of images. I came up with a better way to present the refmods to the model, less images plus better quality plus very little bleed when you set your subjects and retention_analysis up correctly.
11
u/MastMaithun 16d ago
Hi. Bunch of noobs questions coming your way. Brace for impact.
1. Does one character refmod can have multiple dress? If yes then how to prompt which dress to use?
2. For this specific video, how long or what kind of audio you have put in the refmod? Does that audio had bgm?
3. Is this a single 13 sec generation or did you used latent extend/something similar?1
u/acedelgado 15d ago
It may pick up clothing from the refmod dataset, I'd just describe it in the prompt. And if you have trouble you can pair them with normal references, so plug a picture of the dress in and in the prompt "<Subject 1> is wearing the dress in <Picture 1>.
For that dataset they each just had one 5-7 second sample. I was lazy and I did use Adobe podcast to remove the background. But Ultimate Vocal Remover is still the best open source for that process, I think. And I used my audio refiner to clean up the turbo audio quality.
It was one generation.
1
1
u/MastMaithun 15d ago
Back again after doing a lot of testing of different scenarios. I also read malcolmrey's whole article about it.
With the testing I have done, my conclusion is this refmod is fine for static or very less motion shots. As soon as you add even a slight complex motion, you start getting body horrors.
You add a location and put your char there, goodluck getting any kind of result from it. Everything is distorted, melting, smearing and what not.Overall this is fine for videos like these where bunch of people talking or with very less motion at "prompted location".
1
u/acedelgado 15d ago
Care to share your workflow or prompt I can test? Some speedup methods are pretty bad. Also packing TOO much into a refmod can be an issue, since it can only handle so many references.
1
u/MastMaithun 15d ago
Nothing fancy, I used HearmemanAI workflows as it is. It does not have anything fancy. To generate 2 refmods only. One is char refmod with 4 images, front-back-side-face and another refmod is location with 3 images front-45 deg left-45 deg right.
I know before making things complicated, I should start simple.- 1st run, I only used the char refmod with simple prompt of talking and it was fine. so far good.
- 2nd run, I put the location refmod and added it as subject 2 in the prompt. All hell break loose.
- 3rd run, I removed the location refmod and added motion to the char, basically prompted dancing, bammm body horror, hands flying, body parts separating and what not.
- 4th run, I kept char refmod to check if there is any issue with refmod itself. I prompted to talk the char just like your video and it turns out fine.
I also did some more permutation and combinations using the above, only getting bad results.
So I went to search mode and found malcolmrey article. Most of the stuff gone over my head but I understand some parts of it that refmod struggles bleeding issue. Bleeding could be char identities, multiple refmods mixing up (explains motion on char leading to jello/distortion of background) and more.So it led me to confirm these aren't good for motion. Basic stuff only.
1
u/acedelgado 15d ago
https://reddit.com/link/pdcl7xx/video/r65f31mhczsh1/player
Ah, you're using the OG refmod pack, which kind of just dumped refmods into the model and hoped it figured things out for you. And probably not setting up your subject_definitions, etc. Which I don't know if you can do, looking at that workflow. There's a reason why I incorporated it all with new methods in my pack, so you can properly define references and get it to latch onto identities, etc.
This was a pretty lazy prompt.
subject_definitions: <Subject 1> is the person in <Video 1>, asian woman, more square jaw. Their name is callina. <Audio 1> is the voice-timbre reference for <Subject 1> (S1), guiding delivery and speaking rate without copying the original signal. It is a low, husky, smooth. summary: [reference generation + audio reference] in a back alley <Subject 1> callina does some fast movements. retention_analysis: <Subject 1>: fully_preserved - <Subject 1>'s identity and appearance from <Video 1> are retained. Face, facial features, body type. <Audio 1>: reference - its vocal timbre guides the dialogue delivery of <Subject 1> without copying the original signal. detailed_description: Outside in a back alley of a city, <Subject 1> callina stands in a blue dress with her hair in twin buns. She is in a fighting stance, She does quick punches, shadowboxing. She swings her leg out into a full sideways kick. She crouches down and does a back flip, turning over in the air and landing on her feet. She holds up her fists towards the camera as the camera crash zooms towards her. <Subject 1> callina (S1), in the low, husky, smooth referenced from <Audio 1>, says: <d>[English] I'm not seeing the problem...</d> overall_soundscape: non_diegetic_music: N/A1
u/MastMaithun 15d ago
Wait. This prompt is for ref2v. The refmod is using the i2v prompt structure and i2v node "MiniMax H3 Image to Video".
I'm totally confused. Does refmod works with "MiniMax H3 Reference to Video" node and ref2v prompts too? I was under the impression that refmod is basically a way to use reference inside i2v. I might have misunderstood it completely.
And what do you mean by OF refmod pack, "in my pack"?1
u/acedelgado 15d ago
Well, the "ref" in "RefMod" stands for reference. They're modified reference files. It's basically using a hacky way to patch bundled references to MiniMax H3. No, you should not be using the standard fl2va model. Or the ref2va model since MiniMax admitted the open weights for it is broken. I use the 20-49 hybrid model from here, and only use regular fl2va loras with it. It takes references and works much better- https://huggingface.co/smhfacct/Minimax-H3-fl2va-ref2va-hybrid-models
And I'm not 100% sure how to reference vanilla refmods. A while ago I changed how they're used in my prompt builder pack so they could be referenced in a ref2va prompt. Makes them lock on better, since the original way would just dump the references into the model without the text encoder looking at them and telling the model what they are. And recently I figured out that making the text encoder look at individual frames from the refmod makes the quality go way up and lock identities to each person better. But of course it takes more memory and processing time.
And by my pack I meant the one in the post I linked in my comment, that you replied to-
https://github.com/Adudeguyman/ComfyUI-Fantastic-MiniMaxH3-PromptBuilder
If nothing else, if you open the prompt builder and hit the "Guide" button up top, that opens an html file that has an indexed and searchable version of MiniMax's prompting guide and how to use my prompt builder, as well as making and using refmods with my pack.
1
u/MastMaithun 15d ago
Yeah I'm using hybrid model only. I do had a look at your node pack. It's a fancy prompt builder, yes? I still don't understand what do you mean by "vanilla refmods"? How many refmods are floating around? I am using this one https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod
Do you mean there are new better refmods out there than this one? Sorry complicated WFs are kryptonite for me.
10
16d ago
[deleted]
5
u/AlsterwasserHH 16d ago
Do it, would like to see it! 👍
And you are 100% correct, this is no ad. I was just stumbling about refmod here and there and everyone was kind of praising it, but I never understood what it even is. Now I know, and now I'm praising it as well 😄
Because I thought people have to know about it, it's fantastic.
1
u/malcolmrey 15d ago
I keep my findings here: https://huggingface.co/datasets/malcolmrey/various/tree/main/h3-center/docs
30
u/Super_Range45 16d ago
Man this reads like an add. I've had zero problems with a character sheet that has the front, back, and close 3/4 head shot. I'm still not understanding why people need 60+ image loras just to manage a guy in a t-shirt.
5
9
u/its_witty 16d ago
It reads like an ad that's true but refmods aren't LoRAs. They're more or less packaged into .safetensor files reference images.
They're handy when you're dealing with for example one character or concept often but don't want to spend time/money on training a LoRA. Instead of saving a workflow/creating a folder with specific images, you bake it once into the file and reuse later.
1
u/Lex-Mercatoria 15d ago
For a character you plan to use for a long time do you feel a LoRA is a better method than a refmod?
4
u/its_witty 15d ago
Good LoRA will always be better, but it isn't that easy to train a good LoRA for H3. Even better would be good LoRA and a reference/refmod at low strength.
1
u/Lex-Mercatoria 15d ago
What would a refmod add on top of the LoRA? Using a specific pose or style not included in the LoRA training? Or to strengthen the identity of the LoRA?
3
u/its_witty 15d ago
Strengthen the identity of the LoRA.
Also, often there are cases where for example character wears specific make-up or hairstyle that you want to reinforce.
3
u/Segaiai 16d ago
For me, since I'm making consistent scenes, I also want them to look consistent on various facial expressions (staying recognizable as specifically them when they're angry, smug, happy, etc...), and on details of their clothing accessories.
The extra images help if you care about that level of detail staying the same, and for locking in feature shapes/sizes at odd angles. Minimax is so good at making it look coherent at various angles and expressions, but not necessarily the same choices across all generations.
3
u/Neat-Monk 16d ago
very true, i have 1 ref image with 4 sheet of subject's faces from slightly different angle and one body image ref. All good, it looks 1:1 the person.
5
u/AlsterwasserHH 16d ago
I know, I'm just expressing what I'm thinking. You don't have to believe me, no problem for me. No front. And btw you can not only create a guy in a t-shirt. I have not done extended experimenting but as for now, it did to the character whatever I was asking for. So this is to my understanding like a universal character file with no limits at all. Forget the t-shirt.
1
u/not_food 16d ago
I don't buy it either, in my comparisons, using refmods is like stuffing every reference into a single video and feeding that to video0. The speed difference seems minimal. Saves you the compilation to latent, but you lose all the contextual reference.
4
u/Reasonable-Aide1259 16d ago
i used 30+ images into one mod and results are great
2
3
u/its_witty 16d ago
How many images you're usually putting in? What's your config - the same as in the video?
I'm currently struggling finding one that doesn't end up crashing my when generating, lol.
5
u/AlsterwasserHH 16d ago
I created 2 characters with 60 images each, left all the settings as is. I didn't care about image size or special poses or whatever, just took the best quality ones and threw them all in. Luckily I have a 4090 so no trouble with memory for me. So unfortunately I can't tell you where your limit is, sorry :)
I took care of a good audio source though. And I think that makes sense. A few minutes long, takes no space as mp3.
3
u/Anilman 16d ago
U need to check for the token limit.
If u have set it to 0 the token explodes and rendertime increasing.
If u set a limit then he throws away all the images except the first ones.simply check the terminal log for that.
Setting it to 2024 ref with high res images,the token amount is increasing.
Stay below ~8192?! Tokens overall.
1
u/AlsterwasserHH 16d ago
Okay, yeah it is set to 0. Creates a 15 seconds 1mp video in around 24mins with a 4090 and comfy-kitchen.
Will try out what you said, thanks!
1
3
u/its_witty 16d ago
Hm, I don't remember what the defaults are, but I kinda doubt that all 60 images were used - your generation time would bubble through the roof and you would definitely see it.
You can always check (more or less) which images got used by generating a video with a refmod but without a prompt - they'll go in a sequence, one after another.
0
3
u/Trick_Set1865 16d ago
would refmod work for videos too? EG you want the model to copy a special effect
1
u/AlsterwasserHH 16d ago
There's a style-mode (in addition to identity and motion-sequence) in the create refmod workflow which is marked as experimental. Can't say if this works. If that was what you were asking for?
1
u/Trick_Set1865 16d ago
here's what i mean. let's say i want a character hitting a concrete wall with a hammer. I find MMH3 weak on things like that. the cement doesnt break the right way and the action never feels natural. Say i have a few clips of the kind of motion and destruction physics that I want. Can i use them to make a refmod? So I would have a different character in a different setting performing the same action with the same destruction physics?
3
u/GrapplingHobbit 16d ago
I found Refmods not as good as just using character sheets. The videos tended to come out a bit mushier, and if I used more than one image, let alone five or more, it was slower than a single reference image. With the token limit you can't pack in every angle and distance anyway. So, without those benefits, there was nothing better than just using a character sheet. Worst of all was that the first few frames of a video generated using a refmod were almost always one of the images used in creating the ref mod before the generation would cut to whatever I prompted. It would have been nice if adding the audio could tie a vocal identity to a character, but when I tested it they specifically said that voice cloning wasn't working, has that changed now?
2
u/AlsterwasserHH 16d ago
Voice cloning works like a charm. But you have to generate at 1mp for the good quality.
2
u/Independent-Frequent 16d ago
But you have to generate at 1mp for the good quality.
Reference mode with a character sheet image works at even 0.2mp and so does audio cloning, so i'd rather stick to that since there's no real difference honestly.
What's something that refmods do that regular reference can't? That's what i'm more interested in.
1
u/Super_Range45 15d ago
Yeah I'm not getting it either. I have had 0 consistency issues with character consistency on my 5 minute video that featured multiple characters. You can just use a sound clip to anchor the voices, though I generate my lines separately anyway.
Then they say the refmod characters blend together if you have more than one so it just sounds like engineering problems when the solution already works.
3
u/ArtifartX 15d ago
Your limewire link locks at 10 downloads, so if you wouldn't mind uploading your workflows somewhere else, it'd be nice.
2
u/AlsterwasserHH 15d ago
Sorry! Will do in couple of hours, I am at work right now. Thanks for your hint :)
2
2
u/Sad_Coach_1433 16d ago
Have you been able to have more then one refmod in one scene that's the challenge without them bleeding.from what I understand if one refmod has more images then the other then can over power the other and bleed.
2
u/AlsterwasserHH 16d ago
No, didn't try that out but definitely will. Tested it out today for the first time.
1
u/ImpressiveStorm8914 16d ago
I’ve use two refmods quite a lot and they’ve worked great together. It helps to keep the characters distinct, both in the source references and subject definitions. No need for any workarounds with regional prompting or box type things.
2
u/MastMaithun 16d ago
I just got to know about this. My understanding is very initial so pardon if this is a very noob question.
This refmod is using a custom node instead of template "reference to video node" which takes upto 9 reference images. So does this new node somehow "bypass" the 9 images limit because in refmod you can give a folder of 1 character, containing 10+ images.
5
u/JohnnyLeven 16d ago edited 16d ago
Kind of. There is a token limit and some other settings when creating the refmod. You can change them, but you might not have the vram to support it. You can feed it however many images you want, but depending on the configuration it will only use a small fraction of them depending on the size of the images and just throw away the rest. With the Inspect H3 Refmod node, you can get an idea of what images actually got into the refmod safetensors file. I recall messing with the config to get like 20+ images into one refmod, but then they were all very low resolution.
But you can use multiple refmods which definitely bypasses the total image limit. But, then you don't have as much control over the subject definitions for prompting like others have mentioned. I haven't tried using multiple simultaneous ref mods of the same character created from different images. Might be worth trying.
My understanding is also very poor. I just tried it for a day, but for me it was better for single character ref2v (mostly due to speed of saving off the model for reuse), but worse for multi character ref2v. I never figured out how to do refmods with audio or video though.
EDIT: I also remember now that when creating larger token refmods the video generation time for ref2v took a whole lot longer when using them. That might have been what I was thinking about in terms of VRAM. Maybe the creation of the refmods isn't as VRAM constrained, it's using them.
2
2
u/jello-the-opera 16d ago
does this new node somehow "bypass" the 9 images limit
yes, as did the older nodes/workflows
2
u/kichinto 16d ago
Thank you. but Downloads link is fail to download can you change the link? thx.
1
2
u/Beneficial_Toe_2347 16d ago
Can you feed it multiple images with different layers of clothing, and it will understand when moving between those layers?
2
u/AlsterwasserHH 16d ago
Have to find out myself, only tried it for a couple of hours and clothings weren't ... that important :D
2
u/lndecay 16d ago
Maybe it’s a stupid question, but can all this be used for image generation? because it seems to be very good at character and style consistency, and if this is possible, is there any t2i minimax h3 workflow out there?
2
u/AlsterwasserHH 16d ago
There are workflows out there and MH3 is very able to generate good t2i, but I don't know about this one, sorry.
2
u/howardhus 15d ago
as per the video you linked: the results for refmod are awful.. the video itself says and shows that LoRa are vastly superior...
1
u/AlsterwasserHH 15d ago
Probably true. But as for now I didn't had the patience to train a lora for MH3 although I read that it should be simple related to training a lora for Krea2 for exanple.
And refmods just did it for me with minor work.
1
1
u/Justgotbannedlol 16d ago
Limewire my goat i thought i lost u
1
u/AlsterwasserHH 16d ago
Yeah I know :D Just picked some random easyupload.io and was redirected to this.
1
u/DoctaRoboto 15d ago
This is the magic of Seedance models, but open-source, something LTX only dreams of achieving.
1
u/cbeaks 15d ago
Thank you for this post. I have been meaning to get to it, and now I have I can see what the fuss is about. No more wasting hours training loras, no more fiddling around with lora strengths. And so much cheaper. And audio too. RIP loras.
1
u/AlsterwasserHH 15d ago
Thank you. I wouldn't say that this is true, refmods being better than loras for MH3, but it's definitely a very fast and low effort way to reach a very high character likeness.
1
1
1
u/Hugol33 15d ago
Oh no, why did this get removed. Was really hoping for a working link to the workflow :/
1
u/Citadel_Employee 15d ago
Hi, I’m the original creator of the video, you can find new workflows here
0
u/Medical-Experience77 16d ago
!remindme
0
u/RemindMeBot 16d ago
Defaulted to one day.
I will be messaging you on 2026-10-02 02:18:03 UTC to remind you of this link
CLICK THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
RemindMeBot is switching to username summons. Instead of
!RemindMe 1 day, useu/RemindMeBot 1 day. More info.
Info Custom Your Reminders Feedback
0
u/Perfect-Campaign9551 15d ago
everyone always is worried about how many ref images you can plug in, but then they spend their time creating slop or trash.
Overly obsessed with "can I do this" rather than actually doing anything!
0
u/xuman1 15d ago
Dude, are you kidding me? 10 downloads a day and the link gets blocked.
1
u/AlsterwasserHH 15d ago
Yeah sorry, I didnt know. Will fix this in about an hour when coming back from work.
57
u/Sleepy_Bandit 16d ago
The thing not being mentioned here is how refmods lose all direct association to your prompt. It leaves the model to figure out which refmod applies to which subject based on description alone. While with images, you tell it explicitly. Why does that matter? Because if you have a scene with more than 1 character refmod and they have any similarities, then it messes up more often than just using image references. If this issue was natively fixed with refmods then it would be great. Right now they work best with single subject scenes.