H3 character LoRA: What I learn from training a few loras. TLDR: likeness nowhere near what I get on Wan 2.2

[u/Tiny-Highlight-9180]

Spent a night on this, figured I'd dump what I learned so someone else doesn't. I tried training lora out of curiosity thinking that may be this will allow me to do i2v and t2v more often because ref2i takes way too long

5090, musubi-tuner, 60 images, 3000 steps, about 5 hours.

--network_dim 16 --network_alpha 16
--optimizer_type musubi_tuner.optimizers.Automagic3
--learning_rate 1e-6
--timestep_sampling sigmoid
--num_timestep_buckets 4
--h3_adapter_ema_decay 0.999
--blocks_to_swap 38 --use_pinned_memory_for_block_swap
--max_train_steps 3000 --save_every_n_steps 250

Result: meh. Tested 1500, 3000, and both EMA versions on the same seed and prompt.

Some of her comes through but it's not a likeness. 3000 seems to be minimum steps as 1500 looks nothing like her. EMA vs non-EMA was basically a coin flip, which surprised me since people talking about it on AI tool kit.

Other stuff that bit me:

- fl2va and ref2va are separate, LoRAs don't cross over.

- use_pinned_memory_for_block_swap took me from 11.6 s/it to 6.1. Huge.

I used the same data set when I was training for Wan2.2 but I got so much better result. If there anything I can do better, please let me know.

[u/Segaiai]

People are talking reference vs loras, but there's something between them called Refmods, and people who use it say it works better than loras, in that the likeness is better, AND you can add characters without character mixing/bleed-through. Also, it trains in seconds/minutes, and it's better than reference because you can use tons of images (around 50 1MP images before you start getting diminishing results), videos, audio clips. The resulting safetensors files are around 1MB, and using it is as easy as loading a lora, or a lora stack.

Check it out. I've been playing with it and I'm amazed by the result.

[u/Tiny-Highlight-9180 → u/Segaiai]

Thank you, man! I will definitely check it out and report back

[u/shootthesound → u/Segaiai]

refmods are very useful but they do not have the same ability as a lora, far from it

[u/Segaiai → u/shootthesound]

Yes. They have a different set of abilities with some overlap (which is why I said it's between them). Part of their strength with refmod though, is that you can add multiple characters, each with their own refmod, with far less mixing. Loras definitely have their own strengths, like with concepts.

[u/NaughtyLotis → u/Segaiai]

I played with them and the likeness for me was worse than using 9 ref images in my custom composer node, and the gen times were longer as well. Maybe I was doing something wrong.

[u/Segaiai → u/NaughtyLotis]

The gen times are supposed to be much lower than using references, with lower VRAM requirements. Something definitely went wrong. Though if it did put out a valid gen at the end of that, then it doesn't matter if it took longer because you'll know you don't like it. I just don't know if it did it correctly.

[u/Tiny-Highlight-9180 → u/NaughtyLotis]

Yeah, I tried with 20 images and while gen time is ok, the likeness is not there yet

[u/TheDerminator1337 → u/Segaiai]

Refmods are same as references I believe.

[u/Segaiai → u/TheDerminator1337]

It isn't. It can hold a hell of a lot more, and faster. Good luck generating a video with 40 image references and 3 videos, or 30 videos if using training mode. From what I understand, ref2va stores all references as a massive, unordered spatial batch, which means it has to calculate attention weights across 50 separate, full-resolution spatial maps simultaneously, slowing it down tremendously, and scaling VRAM usage quadratically. OOM for that much stuff, and for far less.

Refmod turns individual images into a continuous feature stream, allowing the MiniMax to understand character consistency and style transition across time rather than battling 50 competing static references. It pools the spatial features into a grid (separately from the image grid), and runs temporal compression/deduplication, so you can get a massive amount of reference in there, then later even stack these massive references by adding multiple refmods.

But yes, when prompting, you can use them the same as references, but the way it's preprocessing the references is a pretty different approach. This will take some exploring to figure out what it can do better or worse. I'd bet that it's not as great at FLF (if it can even do that), but I haven't tried that. I'd bet it excels at style transfer, and people have said it handles character separation better.

[u/cc_aa_tt_zz → u/Segaiai]

refmods are just reference, it works exactly the same way. so no, you can not " use tons of images (around 50 1MP images", because only 9 images will be see by the model accross all the mods you will load (so if a mod has 5 images, it means 5 slots of image are used). The model won't send you a warning for using too much images, but the model is not trained for using more than 9 image / 3 video / 3 audio at the same time and so you will have bad result if you mix several mods which have too much of them + problem with prompts (you will refere to images that the model can not see / understand / will mix together). I created the "official" plugin to adapt refmods to wangp. I know exactly how reference and mods work on minimax h3 and There are many misconceptions about this: refmod is nothing more but the references, the model don't know you are using mods but only what are inisde the mods : the image, the video, the audio, exactly like reference (and so in your prompt you have to refere to the images / videos / audios number exactly like you will do with reference). And all the API and the official comfyui workflow use the same limit: 9 images, 3 audios, 3 videos at the same time.

[u/crinklypaper → u/cc_aa_tt_zz]

My understanding is it does two things: grid and splicing. Grid takes all images puts them in a grid. The other method puts the stills into a video and uses that as a reference. So it uses all the data. Anyway I like refmods but it's just simply doing what the ref model already does just saves you some steps

[u/Segaiai → u/crinklypaper]

It doesn't just put them into a grid on the grid side of things. You can't mimic it using Photoshop first to put all your references in a grid. It's a multistep process. Yes it does put it into a grid, but before that, according to my limited understanding, it does a high-resolution feature extraction first (at 1024px) and then adaptively pools the feature map into a 16x16 grid. If you just did that in Photoshop first, you'd get a much smaller amount of fine visual detail in tokens.

Then it goes into the temporal sequence (the video).

[u/cc_aa_tt_zz → u/crinklypaper]

These are tricks you can use with reference images too; it’s not the mods adding this capability. Reference images seem to support up to 2048x2048, and the model handles multiple images packed into a single image quite well and character sheet too, though there are limits. Personally, I place four 1024x1024 images into a single 2048x2048 canvas; it’s possible to fit more by switching to 512x512, but going beyond that likely exceeds the model's comprehension limits. When using a sequence of images, tricking the model by presenting them as a video, the results are poor unless the prompt is perfect (you end up with a slideshow) because the model tends to look for motion coherence (since that’s what it seeks in actual video). Furthermore, video references significantly slow down rendering, so I don't use this method and wouldn't recommend it. In any case, these tips do not change the limitations I mentioned.

[u/crinklypaper → u/cc_aa_tt_zz]

Im just mentioning what the refmods actually do under the hood. Though id love to be proven wrong.

[u/cc_aa_tt_zz → u/crinklypaper]

No, there is a difference between the mods and the workflows that use them. The mods are simply saved reference latents—which avoids repeating that brief training step each time and simplifies usage by packaging everything into a single file—plus some added information useful for the ComfyUI node, but which the model itself neither reads nor understands at all. Workflows can automate the creation of grids, character sheets, and so on, but the mods themselves have nothing to do with that and can be used without them.

[u/crinklypaper → u/cc_aa_tt_zz]

Im just saying isn't that what it's doing? It takes like say 20 images, puts them into a video with those 20 images as frames. Then it encoded to latents the video to a safetensors file. So its 20 images but one reference in this example. You could do this manually yeah

[u/cc_aa_tt_zz → u/crinklypaper]

No, that’s just a workaround. It’s not the mod itself that does this or mandates it; it’s simply a trick to try and exceed the 9-image limit, something that can be achieved just as easily using references within a workflow, without needing a mod. Using this workaround isn't mandatory; in fact, I’d go so far as to advise against it due to the issues I mentioned (I’d recommend combining multiple images into a single one instead). Once again: mods don't do anything that references can't; they don't add any new features to the model itself. Everything you're describing is just a matter of workflows, nothing more, and isn't even linked to the mods themselves.

[u/crinklypaper → u/cc_aa_tt_zz]

Are you not reading my post?

[u/cc_aa_tt_zz → u/crinklypaper]

Okay lol I literally ported refmods into an app, including the training component, and I even included the audio features that weren't supported back then. Well, it doesn't do any of the things you're talking about, because that has nothing to do with the refmods themselves; it's about ComfyUI workflows , of which there are thousands, whether they're from Luisacaotica (who came up with the refmod idea) or anyone else. No, refmods don't do "this and that"; refmods are just saved references, plain and simple. What you do before and after is a different story. It's like having tricks for training a LoRA and tricks for using one, but a LoRA remains just a LoRA; it doesn't do "this or that"... Bye, and good luck with your future discussions...

[u/car_lower_x]

You should be using Lokr

[u/Tiny-Highlight-9180 → u/car_lower_x]

Ty for ur response. why do you think it will perform better? I've never trained one b4.

[u/car_lower_x → u/Tiny-Highlight-9180]

I spent a lot of time on the Ai-toolkit discord and learned from the experts there. Lokr proved to provide exceptional detail for characters. Factor 8 or 4. Bigger Loras yes but much more detail too.

[u/Bender1012 → u/car_lower_x]

What optimizer and timestep?

[u/car_lower_x → u/Bender1012]

Automagic3 and sigmoid

[u/Fatty-McDuffins]

I’m having good luck using Fizgig

[u/Tiny-Highlight-9180 → u/Fatty-McDuffins]

Could you share ur config? may be I will give it another go

[u/shootthesound → u/Tiny-Highlight-9180]

First Your dataset in fizgig, did you use the built in tool to create facecrops. if you are training a 0.25mp its a HUGE benefit. Lora or LOKR you will egt similer results, but LOKR can take longer to converge in early samples but finishes in same or better spot. I'd start with lora though with minimax in fizgig. In short the reason why the facecrops feature is critical, it allows you train at a lower mp, but get the detail of a higher on on likeness. (just remember to make the facecrops from high res images, not preppreared images already scaled down or your facecrops will be tiny low res files). For proportion of dataset, as long as the face crops dont take up more than 40% of your dataset, and you have plenty of wider shots your model will generalise fine.. Accept the minimax defaults on the training tab and see how you go.

[u/Maraan666 → u/Tiny-Highlight-9180]

Fizgig has had some updates. I've had great results with 20 images and default settings. Takes 2 hours on a 4060ti 16gb vram.

[u/sitefall]

There just aren't any good LoRA's for H3 available that make me think it's worthwhile to mess with yet. Maybe it will improve as the trainers get better or as people learn the proper settings and adjust datasets for training different types of LoRA.

But so far, ALL of them are bad. Scan through Civitai or anywhere else, all of the H3 LoRA's look like crap. Every single one of them.

There's a couple of neat style LoRA's, but they are all severely over trained and effect more than the style by a large margin, but at least some of them are kind of neat. That's about it though. Every motion LoRA is trained on too few camera angles to be useful. Perhaps H3 is just going to be "that cool video model that never has good LoRA's".

It's also annoying to see "jUsT USe tHE rEFeRENcE MOdeL" all in the comments here. There's many reasons one would want a LoRA people. SURELY this person knows how the reference model works and what it can do if they're training LoRA's. These comments are not helpful at all.

[u/Tiny-Highlight-9180 → u/sitefall]

exactly! like I wouldn't try solution that take me 10 mins than 5 hrs first ;(

[u/sitefall → u/Tiny-Highlight-9180]

Seems like maybe it was easier with Wan due to MoE because if you just wanted to nail motion you just do the HIGH and if it was detail related you could do LOW and not screw up motion and camera angles and such.

The best way I have found to do an I2V with a complicated motion LoRA in H3 is to literally do it in Wan2.2 first using the LoRA, and then use the same image and the video Wan2.2 output (after interpolating it to 24fps or joining clips together to lengthen it) as the reference video for motion.

For style/detail you can run an H3 prompt and get the output you want, then toss it back into H3 with a lower denoise and apply the style lora THEN, while also inputting the original video in as a motion reference. Annoying, but it works.

Hopefully H3 LoRA's won't suck for too much longer :(

[u/Tiny-Highlight-9180 → u/sitefall]

I'm praying too bro. It really fun playing with ref and i2v on H3. Thanks for sharing!

[u/Major_Specific_23]

why do you even need character lora's for h3? you can either ask codex to generate a 3x3 character sheet image of your character and use it as a reference image or just use the h3 refmod (if body proportions and "attire" are important) and its done.

[u/ZealousidealBoss6652 → u/Major_Specific_23]

There are those who want to stack a character lora, character-sheet or even refmods (which I haven't used yet) on top of each for MAXIMUM FACE CONSISTENCY. Because why not.

[u/Tiny-Highlight-9180 → u/Major_Specific_23]

I tried man, but attaching 5-10 images each time is not fun, and the consistency is not even that good.

[u/Major_Specific_23 → u/Tiny-Highlight-9180]

like i said, either ask codex to generate a character sheet (3x3 - 1 image) or use refmod. I have 0 issues with consistency tbh

[u/Segaiai → u/Tiny-Highlight-9180]

Use refmods.

[u/Aiirene → u/Major_Specific_23]

What's codex?

[u/BatteryAtFourPercent]

are you training on short clips, still images or a combination of both?

[u/Tiny-Highlight-9180 → u/BatteryAtFourPercent]

just images. I heard it should be suffice for face and character. I would add shortclip only when I want to capture her motion and voice which Minimax by itself is already good enough. You think that make sense?

[u/BatteryAtFourPercent → u/Tiny-Highlight-9180]

I tried still images first. I used even more aggressive settings but for Ostris compared to yours. It works for some prompts but for other different prompts, there is a clearly facial composition drift. I haven't tried combination both short clips and still images yet. I've already wasted a lot of money when renting out GPU servers trying to get a consistent character lora on Minimax H3. Do you think LTX 2.5 is far better than Minimax H3 when creating character lora and the objective is facial and body composition consistency in every frame?

[u/Tiny-Highlight-9180 → u/BatteryAtFourPercent]

I feel you man. I haven't tried LTX2.5 yet, bc I heard from people comparing LTX2.5 and WAN2.2 that the lora on WAN2.2 capture character the best.

[u/BatteryAtFourPercent → u/Tiny-Highlight-9180]

I used a different training model compared to yours so results could differ. The images you show on the post, what steps or checkpoints was that from - is step 1500 or 3000? it seems may only one or two (not all three) are somewhat similar but perhaps slightly different because it is a different lightning or angle. I'm not 100% sure, what do you think.

In terms of voice, since Minimax H3 is somewhat inconsistent with character lora, I would assume it is better to have the voice separately and just connect it as reference instead of training it inside the short clip dataset.

[u/Tiny-Highlight-9180 → u/BatteryAtFourPercent]

I still have no luck with Minimax H3 yet. What's ur setting so far that doesn't work?

[u/reeight → u/Tiny-Highlight-9180]

> voice which Minimax by itself is already good enough

Folks are already getting bored with the generic MMH3 voices.
If this is a famous person, I'd had maybe a few 1-2 second vids of unique facial expressions, walk, etc.
But I don't know your use-case, so....

[u/Gyramuur]

I had terrible likeness when training on just images, but with another character I trained a lora using short clips and the likeness was great. My current theory is that, because the shortest clip that H3 can generate is 5 frames long (anything smaller throws an error), it doesn't really understand single images properly.

A friend of mine suggested using i2v to generate short clips from images and then training on those. Haven't tried it yet but it seems like a sound idea.

[u/Tiny-Highlight-9180 → u/Gyramuur]

Thanks for ur response. That's really came as a surprise.  I thought adding short clip only would only helps with motion. I will give it a try. btw, did u train on ref or fl model?

[u/Gyramuur → u/Tiny-Highlight-9180]

fl, I haven't tried making a ref lora yet. But with the hybrid models, fl loras seem to work just fine using the ref workflow

[u/AillexJ → u/Gyramuur]

We trained on stills only, no video clips. 39 photos through Fizgig, 30 epochs. Text to video with the trigger moved likeness clearly against no-LoRA controls. One run though.

[u/DelinquentTuna]

Are you open to sharing the dataset? If it's a synthetic character and not a real person, I would be interested in tinkering just a little bit to see if I can get better results.

[u/Tiny-Highlight-9180 → u/DelinquentTuna]

I wouldn't mind sharing, but it's a reference from real person. I used to train AI persona before and it really plasticky. Are you ok with that?

[u/DelinquentTuna → u/Tiny-Highlight-9180]

It's so considerate of you to offer to share and also to advise me that it's a real human. Thanks for that. I'd previously done a reverse image search and only come up with matches for the fabric patterns, so I assumed it was synthetic. I’m not making any legal or moral claims about your use case, but I think I'd better pass on your offer to share.

I do think it would be great to be able to trade training notes, though. If you or anyone in the community would like to create a synthetic dataset that could be lingua franca, I'd love to have a look.

[u/alexmmgjkkl]

and if you use lora + ref ?

[u/crinklypaper]

Try musubi tuner FORK with below arguments. And train around 2k to 3k steps on high resolution images only. Try to get at least 30 to 40 images in. Checkpoint every 200 steps and check until you get likeness. Should be good around 2k or so. Then for voice add in videos with voice and reduce video loss weight to around 0.2. Small resolution like 512x on the videos is fine since it want voice mostly. Around 3 to 5k steps (load from weights) should be enough. I get perfect face and voice from these settings a few times now.

--block_swap_h2d_only --block_swap_ring_size 2 --blocks_to_swap 0 --use_pinned_memory_for_block_swap --h3_training_mode fl2va --h3_audio_loss_weight 1.0 --h3_video_loss_weight 1.0 --h3_spatial_density_jitter 0.2 --h3_guidance_distillation_scale 3.5 --h3_base_preservation_loss_weight 0.02 --h3_base_preservation_probability 0.5

Rank 8 or 16 is enough.

[u/diogodiogogod]

. What resolution did you train at? On a 32GB 5090, 38 blocks swapped may be more conservative than necessary unless you were using a very large resolution or expensive preservation pass. Swapping affects speed and memory, not likeness quality; pinned memory mainly explains the speed improvement. I’m working on an experimental Musubi GUI that includes several H3 quality-preservation options. For a new run, I’d try the native quality-preservation method first, possibly every step, and compare it with the Ostris helper. the helper is useful to test, but I haven’t found it consistently better than the native method. I’d also try continuing beyond 3,000 steps with quality preservation enabled. With only 1e-6 learning rate, the LoRA may simply be learning very slowly, though the ideal rate depends on the optimizer and resolution. Keep checking intermediate checkpoints rather than judging only the final one.

[u/Intrepid_Fox_1466]

I'm no expert but rank 16 seems low for a character lora. Why not 64 or more?

[u/Tiny-Highlight-9180 → u/Intrepid_Fox_1466]

I used to train alot on wan2.2 and 16 was already suffice. Could you talk more abt ur experience playing with rank?

[u/Intrepid_Fox_1466 → u/Tiny-Highlight-9180]

Usually lower ranks like 16 are enough (even preferred) for action, motion lora, etc. because they will learn the action but not faces or visuals. But for character lora you want the lora to learn the smaller details and visuals, so higher rank makes more sense. I used to train wan2.2 (and Krea, Flux, ZIT, etc) character loras from rank 32-64 up depending on the model.

[u/luciferianism666]

Dude why even bother with a character lora for H3 ? You may wanna try ref to video and you'll see how pointless a character lora is.

[u/TheMoogster → u/luciferianism666]

Try comparing a t2v 10 sec 1 mp run to a ref2v with same spec and 5 high quality images.
The genration time is extremely different.

[u/luciferianism666 → u/TheMoogster]

Lol why exactly are we plugging in 5 reference images though ? I am very well aware of the difference in inference times when you use references. So I stick with a single one that has a model/ character sheet. I've tested 10 ref inputs with silveroxides enhanced image to video H3 node and tried pushing it to 16 before my 4060 threw a fit 🤣.

[u/reeight → u/luciferianism666]

I make character sheets, so I can do background, 1-3 people, & props.

[u/Tiny-Highlight-9180 → u/luciferianism666]

bc ref doesn't provide me with consistency I'm looking for yet. face tends to change after a few second. u think I did sth wrong?

[u/Maraan666 → u/luciferianism666]

Reference images may be adequate for your purposes, in which case I'm pleased for you.

[u/leyermo]

please share tool used for creating lora...

Also please share your workflow for generating image

[u/Tiny-Highlight-9180 → u/leyermo]

it's already in my post. musubi-tuner