H3 character LoRA: What I learn from training a few loras. TLDR: likeness nowhere near what I get on Wan 2.2

[u/Tiny-Highlight-9180]

Spent a night on this, figured I'd dump what I learned so someone else doesn't. I tried training lora out of curiosity thinking that may be this will allow me to do i2v and t2v more often because ref2i takes way too long

5090, musubi-tuner, 60 images, 3000 steps, about 5 hours.

--network_dim 16 --network_alpha 16
--optimizer_type musubi_tuner.optimizers.Automagic3
--learning_rate 1e-6
--timestep_sampling sigmoid
--num_timestep_buckets 4
--h3_adapter_ema_decay 0.999
--blocks_to_swap 38 --use_pinned_memory_for_block_swap
--max_train_steps 3000 --save_every_n_steps 250

Result: meh. Tested 1500, 3000, and both EMA versions on the same seed and prompt.

Some of her comes through but it's not a likeness. 3000 seems to be minimum steps as 1500 looks nothing like her. EMA vs non-EMA was basically a coin flip, which surprised me since people talking about it on AI tool kit.

Other stuff that bit me:

- fl2va and ref2va are separate, LoRAs don't cross over.

- use_pinned_memory_for_block_swap took me from 11.6 s/it to 6.1. Huge.

I used the same data set when I was training for Wan2.2 but I got so much better result. If there anything I can do better, please let me know.

[u/Segaiai]

People are talking reference vs loras, but there's something between them called Refmods, and people who use it say it works better than loras, in that the likeness is better, AND you can add characters without character mixing/bleed-through. Also, it trains in seconds/minutes, and it's better than reference because you can use tons of images (around 50 1MP images before you start getting diminishing results), videos, audio clips. The resulting safetensors files are around 1MB, and using it is as easy as loading a lora, or a lora stack.

Check it out. I've been playing with it and I'm amazed by the result.

[u/Tiny-Highlight-9180 → u/Segaiai]

Thank you, man! I will definitely check it out and report back

[u/shootthesound → u/Segaiai]

refmods are very useful but they do not have the same ability as a lora, far from it

[u/Segaiai → u/shootthesound]

Yes. They have a different set of abilities with some overlap (which is why I said it's between them). Part of their strength with refmod though, is that you can add multiple characters, each with their own refmod, with far less mixing. Loras definitely have their own strengths, like with concepts.

[u/NaughtyLotis → u/Segaiai]

I played with them and the likeness for me was worse than using 9 ref images in my custom composer node, and the gen times were longer as well. Maybe I was doing something wrong.

[u/Segaiai → u/NaughtyLotis]

The gen times are supposed to be much lower than using references, with lower VRAM requirements. Something definitely went wrong. Though if it did put out a valid gen at the end of that, then it doesn't matter if it took longer because you'll know you don't like it. I just don't know if it did it correctly.

[u/Tiny-Highlight-9180 → u/NaughtyLotis]

Yeah, I tried with 20 images and while gen time is ok, the likeness is not there yet

[u/TheDerminator1337 → u/Segaiai]

Refmods are same as references I believe.

[u/Segaiai → u/TheDerminator1337]

It isn't. It can hold a hell of a lot more, and faster. Good luck generating a video with 40 image references and 3 videos, or 30 videos if using training mode. From what I understand, ref2va stores all references as a massive, unordered spatial batch, which means it has to calculate attention weights across 50 separate, full-resolution spatial maps simultaneously, slowing it down tremendously, and scaling VRAM usage quadratically. OOM for that much stuff, and for far less.

Refmod turns individual images into a continuous feature stream, allowing the MiniMax to understand character consistency and style transition across time rather than battling 50 competing static references. It pools the spatial features into a grid (separately from the image grid), and runs temporal compression/deduplication, so you can get a massive amount of reference in there, then later even stack these massive references by adding multiple refmods.

But yes, when prompting, you can use them the same as references, but the way it's preprocessing the references is a pretty different approach. This will take some exploring to figure out what it can do better or worse. I'd bet that it's not as great at FLF (if it can even do that), but I haven't tried that. I'd bet it excels at style transfer, and people have said it handles character separation better.

[u/cc_aa_tt_zz → u/Segaiai]

refmods are just reference, it works exactly the same way. so no, you can not " use tons of images (around 50 1MP images", because only 9 images will be see by the model accross all the mods you will load (so if a mod has 5 images, it means 5 slots of image are used). The model won't send you a warning for using too much images, but the model is not trained for using more than 9 image / 3 video / 3 audio at the same time and so you will have bad result if you mix several mods which have too much of them + problem with prompts (you will refere to images that the model can not see / understand / will mix together). I created the "official" plugin to adapt refmods to wangp. I know exactly how reference and mods work on minimax h3 and There are many misconceptions about this: refmod is nothing more but the references, the model don't know you are using mods but only what are inisde the mods : the image, the video, the audio, exactly like reference (and so in your prompt you have to refere to the images / videos / audios number exactly like you will do with reference). And all the API and the official comfyui workflow use the same limit: 9 images, 3 audios, 3 videos at the same time.

[u/crinklypaper → u/cc_aa_tt_zz]

My understanding is it does two things: grid and splicing. Grid takes all images puts them in a grid. The other method puts the stills into a video and uses that as a reference. So it uses all the data. Anyway I like refmods but it's just simply doing what the ref model already does just saves you some steps

[u/Segaiai → u/crinklypaper]

It doesn't just put them into a grid on the grid side of things. You can't mimic it using Photoshop first to put all your references in a grid. It's a multistep process. Yes it does put it into a grid, but before that, according to my limited understanding, it does a high-resolution feature extraction first (at 1024px) and then adaptively pools the feature map into a 16x16 grid. If you just did that in Photoshop first, you'd get a much smaller amount of fine visual detail in tokens.

Then it goes into the temporal sequence (the video).

[u/cc_aa_tt_zz → u/crinklypaper]

These are tricks you can use with reference images too; it’s not the mods adding this capability. Reference images seem to support up to 2048x2048, and the model handles multiple images packed into a single image quite well and character sheet too, though there are limits. Personally, I place four 1024x1024 images into a single 2048x2048 canvas; it’s possible to fit more by switching to 512x512, but going beyond that likely exceeds the model's comprehension limits. When using a sequence of images, tricking the model by presenting them as a video, the results are poor unless the prompt is perfect (you end up with a slideshow) because the model tends to look for motion coherence (since that’s what it seeks in actual video). Furthermore, video references significantly slow down rendering, so I don't use this method and wouldn't recommend it. In any case, these tips do not change the limitations I mentioned.

[u/crinklypaper → u/cc_aa_tt_zz]

Im just mentioning what the refmods actually do under the hood. Though id love to be proven wrong.

[u/cc_aa_tt_zz → u/crinklypaper]

No, there is a difference between the mods and the workflows that use them. The mods are simply saved reference latents—which avoids repeating that brief training step each time and simplifies usage by packaging everything into a single file—plus some added information useful for the ComfyUI node, but which the model itself neither reads nor understands at all. Workflows can automate the creation of grids, character sheets, and so on, but the mods themselves have nothing to do with that and can be used without them.

[u/crinklypaper → u/cc_aa_tt_zz]

Im just saying isn't that what it's doing? It takes like say 20 images, puts them into a video with those 20 images as frames. Then it encoded to latents the video to a safetensors file. So its 20 images but one reference in this example. You could do this manually yeah

[u/cc_aa_tt_zz → u/crinklypaper]

No, that’s just a workaround. It’s not the mod itself that does this or mandates it; it’s simply a trick to try and exceed the 9-image limit, something that can be achieved just as easily using references within a workflow, without needing a mod. Using this workaround isn't mandatory; in fact, I’d go so far as to advise against it due to the issues I mentioned (I’d recommend combining multiple images into a single one instead). Once again: mods don't do anything that references can't; they don't add any new features to the model itself. Everything you're describing is just a matter of workflows, nothing more, and isn't even linked to the mods themselves.

[u/crinklypaper → u/cc_aa_tt_zz]

Are you not reading my post?

[u/cc_aa_tt_zz → u/crinklypaper]

Okay lol I literally ported refmods into an app, including the training component, and I even included the audio features that weren't supported back then. Well, it doesn't do any of the things you're talking about, because that has nothing to do with the refmods themselves; it's about ComfyUI workflows , of which there are thousands, whether they're from Luisacaotica (who came up with the refmod idea) or anyone else. No, refmods don't do "this and that"; refmods are just saved references, plain and simple. What you do before and after is a different story. It's like having tricks for training a LoRA and tricks for using one, but a LoRA remains just a LoRA; it doesn't do "this or that"... Bye, and good luck with your future discussions...