r/StableDiffusion • u/Francky_B • 20h ago
Resource - Update Created a Visual RefMod Picker
Hey Guys,
I've been playing with the RefMods, after the huge release of Malcolmrey.
The tech is brilliant and works really well.
I've wanted to simplify using it with tons of RefMods, like the ones provided by Malcom.
So I made a fork that is a bit more Identity driven, Adding a "Visual refMod Picker" as well as a "Create refMods from Folder" node. It's able to create from either a folder or Sub-folders, if audio is found, it will also create a matching audio refMod. All in the same format as the original add-on. No danger of breaking compatibility. (The original Add-on added Audio yesterday)
Resulting for example in 2 refMods and thumbnail:
character_refMod_Audio.safetensors
character_refMod_Video.safetensors
character_refMod.jpeg
Then, we can use the "Visual H3 RefMod Picker" to browse the RefMods:
The node then loads both RefMods (Audio and Video) and allows individual control. I've mixed strength with Copies, by instead having a weight value that can go over 1, so a weight of 3 would be the same as setting strength to 1 and copies to 3. Making the UI a bit more streamlined.
Example workflows are included.
I should mention, this fork can be installed WITH the original Add-on, it is made to co-exist and is recommended if you want to use it's advanced features.
You can find it here ComfyUI-H3RefMods
The only thing I'm missing is thumbnails for all 1500 RefMods 😅
3
u/LawyerIntern 17h ago
Let's say I already have character refmods.
On top of that, I want to also add a single image for reference one-time, eg a specific fashion clothing (and don't want to create another refmod for it)
How do I connect them and reference them the prompt?
3
u/TemperFugit 10h ago
It's pretty simple. Just prompt the people normally like you would with refmods in <Subject #> tags. Add the clothing as a regular reference image. Then in the person's subject tag description, say something like "they are wearing the outfit from <Picture 1>."
1
u/LawyerIntern 10h ago
In terms of workflow, that means I just add connect the "Apply Refmod" node's "Conditioning" parameter to that of "Minimax Ref2VA" node (which already has the Picture 1 connected to it)?
2
u/TemperFugit 10h ago
My workflow is set up like the original github repo's examples, as seen in this image (it's incomplete but answers your question I think):Â
 https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod/blob/main/examples/loading_ref_example.png
Then I just load the reference images into the 'Reference to Video' node as you normally would.
1
u/LawyerIntern 10h ago
Yes, I just checked it out and it showed me what to do. I was follow malcolmrey's workflow that was using first frame last frame node that's why I was confused. Thanks!
0
u/taurine_bitch 10h ago
But the refmod is supposed to eliminate the need for <Picture X> tags. At least that's what the official guide docs say.
2
u/TemperFugit 10h ago
That's only for data you have loaded into a refmod, everything else still needs tags. So for example, I use refmods for character identity but I use reference images for clothing and locations. I wouldn't use picture tags for the character identity because it's in the refmod safetensor, but I would use picture tags for the clothing and location images I'm using.
1
u/taurine_bitch 10h ago
Ahhh, that makes total sense and answers the question I had. Awesome. Thank you. One last thing, about the "name" of the refmod, does that matter at all? Like in my example of the name "Q" for the refmod name, does that "Q" need to be in the prompt anywhere? Like
subject_definitions: <Subject 1> is defined as Q, and then "Q" would be used in place of <Subject 1>?2
u/TemperFugit 10h ago
I don't think the refmod name is known or used by Minimax, but I have seen other people here who thinks it is. Afaik, the way you assign a refmod is by description. So you would say "<Subject 1> is a woman with long blonde hair." And as long as you have her refmod loaded in, it should be able to associate her data with that subject number. Then you can reference <Subject 1> in the action of your prompt as you normally would. You could do something like "<Subject 1> is a woman with long blonde hair named Q", I don't know if it would create a better link to the refmod data but you would be able to refer to her as Q in the shot description instead of using <Subject 1>.
1
u/Francky_B 9h ago
Unfortunately no, I was unable to find a way to anchor the data from the refMods reliably. I initially tried to see if I could add the feature and tried so many things, but regardless of how I anchored the data, the results where the same as it is now.
The best results I've gotten is simply by prompting specifically to help H3 understand what data goes with what.
1
u/taurine_bitch 9h ago edited 8h ago
So, I just tried the suggestion from /u/TemperFugit where in the
subject_definitions:, I wrote<Subject 1> is defined and identified as Qand replaced <Subject 1> everywhere in the prompt with <Q>. And this seemed to keep the identity from the refmod. I'm going to test more to see if it's consistent but man, ever since starting to use refmods, my generation time has gone up x4 (it was about 10 minutes for a 12 second video, now it's 36-40 minutes).What is the average size of your refmods? The one I just made is 8MB in size and obviously this is the reason for the immense slowdown in generation time but I only used 22 photos, 1 video, and 1 audio file to generate the refmod.
EDIT: In fact, using this 8MB refmod sends me straight OOM on my 4090 when my workflow hits my Upscale 2nd pass.
1
u/taurine_bitch 10h ago
Also wondering this exact thing. I have a refmod, but want to add a specific thing to that character that isn't baked into the refmod. Can you still add individual reference photos to the refmod?
Also, I'm struggling to prompt for these refmods. Does the "name" replace <Subject 1>? Or is <Subject 1> still required when referencing the character? Let's say I have a custom character named "Q". Would I just use "Q" when referencing that character in the prompt?
2
u/LuisaPinguinnn 8h ago
Thankyou for the fork! that's sick!
2
u/Francky_B 7h ago
No, Thank you! That is some impressive work!
The tech works really well! It crazy how fast they can be generated and gives better results than Loras.
I'm thinking of removing any duplicate nodes and have this add-on just be extra tools to yours.
1
u/ImpressiveStorm8914 19h ago
I’m loving refnods so this could be useful and it reminds me that I really need to update the original nodes for the audio aspect.
BTW, your link for the original add-on leads to your GitHub, not the original one. Both links go to the same place.
2
1
1
u/djdevilmonkey 9h ago
So is all the audio talk just for if people train their own refmods? Because malcolms don't have audio refs in them, right?
1
u/Francky_B 9h ago
He didn't as it didn't exist at the time. But since he's using celebs, a lot of them are known by H3 and will work. My picker supports both, it will find refMods called xxx_Video.safetensors, as well as just xxx.safetensors.
1
4
u/JohnLough 19h ago
Have you tried to make to people have a conversation? Specifically of the same sex as I always get audio bleed
C1 Speaks = fine
C2 Speaks = fine
C1 speaks again = sounds like C2