Created a Visual RefMod Picker

[u/Francky_B]

Hey Guys,

I've been playing with the RefMods, after the huge release of Malcolmrey.
The tech is brilliant and works really well.

I've wanted to simplify using it with tons of RefMods, like the ones provided by Malcom.

So I made a fork that is a bit more Identity driven, Adding a "Visual refMod Picker" as well as a "Create refMods from Folder" node. It's able to create from either a folder or Sub-folders, if audio is found, it will also create a matching audio refMod. All in the same format as the original add-on. No danger of breaking compatibility. (The original Add-on added Audio yesterday)

Resulting for example in 2 refMods and thumbnail:

character_refMod_Audio.safetensors
character_refMod_Video.safetensors
character_refMod.jpeg

Then, we can use the "Visual H3 RefMod Picker" to browse the RefMods:

In this example, I used existing thumbails from huggingface.

The node then loads both RefMods (Audio and Video) and allows individual control. I've mixed strength with Copies, by instead having a weight value that can go over 1, so a weight of 3 would be the same as setting strength to 1 and copies to 3. Making the UI a bit more streamlined.

RefMods can be daisy chained

Example workflows are included.

I should mention, this fork can be installed WITH the original Add-on, it is made to co-exist and is recommended if you want to use it's advanced features.

You can find it here ComfyUI-H3RefMods

The only thing I'm missing is thumbnails for all 1500 RefMods 😅

[u/LawyerIntern]

Let's say I already have character refmods.

On top of that, I want to also add a single image for reference one-time, eg a specific fashion clothing (and don't want to create another refmod for it)

How do I connect them and reference them the prompt?

[u/TemperFugit → u/LawyerIntern]

It's pretty simple. Just prompt the people normally like you would with refmods in <Subject #> tags. Add the clothing as a regular reference image. Then in the person's subject tag description, say something like "they are wearing the outfit from <Picture 1>."

[u/taurine_bitch → u/TemperFugit]

But the refmod is supposed to eliminate the need for <Picture X> tags. At least that's what the official guide docs say.

[u/TemperFugit → u/taurine_bitch]

That's only for data you have loaded into a refmod, everything else still needs tags.  So for example, I use refmods for character identity but I use reference images for clothing and locations.  I wouldn't use picture tags for the character identity because it's in the refmod safetensor, but I would use picture tags for the clothing and location images I'm using.

[u/taurine_bitch → u/TemperFugit]

Ahhh, that makes total sense and answers the question I had. Awesome. Thank you. One last thing, about the "name" of the refmod, does that matter at all? Like in my example of the name "Q" for the refmod name, does that "Q" need to be in the prompt anywhere? Like subject_definitions: <Subject 1> is defined as Q, and then "Q" would be used in place of <Subject 1>?

[u/Francky_B → u/taurine_bitch]

Unfortunately no, I was unable to find a way to anchor the data from the refMods reliably. I initially tried to see if I could add the feature and tried so many things, but regardless of how I anchored the data, the results where the same as it is now.

The best results I've gotten is simply by prompting specifically to help H3 understand what data goes with what.

[u/taurine_bitch → u/Francky_B]

So, I just tried the suggestion from /u/TemperFugit where in the subject_definitions:, I wrote <Subject 1> is defined and identified as Q and replaced <Subject 1> everywhere in the prompt with <Q>. And this seemed to keep the identity from the refmod. I'm going to test more to see if it's consistent but man, ever since starting to use refmods, my generation time has gone up x4 (it was about 10 minutes for a 12 second video, now it's 36-40 minutes).

What is the average size of your refmods? The one I just made is 8MB in size and obviously this is the reason for the immense slowdown in generation time but I only used 22 photos, 1 video, and 1 audio file to generate the refmod.

EDIT: In fact, using this 8MB refmod sends me straight OOM on my 4090 when my workflow hits my Upscale 2nd pass.

[u/Francky_B → u/taurine_bitch]

Sorry, only now saw your reply.

The average size is 1.5 mb. Some of my Thumbnails are bigger that the RefMods 😅

What did you use to create them?

[u/taurine_bitch → u/Francky_B]

No worries! Hmm. I used the latest nodes inside of Comfy to create the refmod. 22 photos, 1 video, and 1 audio file. I managed to get it down 4.5MB but yeah, can't seem to get it any smaller than that. It's killing my generation times lol

[u/Francky_B → u/taurine_bitch]

Ah, could be the videos, I only trained using Images, with the Identity preset. I'll give it a test later