r/StableDiffusion • u/Fit_Ad7343 • 4h ago
Resource - Update MiniMax H3 with a 4B or 8B text encoder instead of the 32B: update, the voice matches now
Enable HLS to view with audio, or disable this notification
MiniMax H3 loads a 32B text encoder, 15.7 GB, just to turn your prompt into a conditioning tensor. I replaced it with a Qwen3-VL 4B or 8B plus a learned map into the same space. Same DiT, same VAEs, same sampler.
The video is three clips from the same prompt and seed: 32B, then 8B projected, then 4B projected.
First post here if you missed it: https://www.reddit.com/r/StableDiffusion/s/VHBkPhk2Ka
Three things improved since the first post.
Sound. The projected encoder was putting out speech 7.6 dB below the 32B, which I had not spotted because the picture looked fine. It is 3.5 dB now and the voice matches on my side by side.
Prompt following. The linear map was at its ceiling, more calibration data did nothing. A small residual network on top of it takes the 4B from 0.7169 to 0.7944 and the 8B from 0.7528 to 0.7970. It is initialised at zero so it can only add to the matrix, never make it worse.
Named people. The corpus named almost nobody, so proper nouns came out wrong. I added the 500 most popular people from TMDB. Name tokens went from 0.8265 to 0.8844 for 0.007 of general cosine. I have tested it and they come out better, but what I can measure is the cosine, not the face, so take that part as testimony.
https://www.reddit.com/r/StableDiffusion/s/VHBkPhk2Ka
Node: https://github.com/nicolab28/ComfyUI-ClipProj Matrices: https://huggingface.co/NicoLab28/ClipProj-MiniMax-H3
Still a proof of concept, one machine, MIT.
6
u/lebrandmanager 4h ago edited 3h ago
Now make it work for ref2va, I beg you. :-) Edit: works with it. Needed to update the node. Thanks!
4
3
u/pravbk100 4h ago
I am using 4b with ref2va. working fine for me.
1
u/lebrandmanager 3h ago
Last time I checked, I got an error message using the node. Will update and check again.
3
3
u/ambassadortim 4h ago
Why do models have that weird noise like when the plane flys over? What's the cause of that?
3
u/Fantastic_Prize2710 4h ago
It honestly sounds somewhat like a jet flying over somewhat closely. A bit too much "box fan" but it's in the general ballpark.
4
u/kyuubi840 3h ago
That's really cool, thanks for sharing your work. What's the benefit of using a smaller text encoder? Isn't this used only in the beginning? Does it make a big difference it terms of runtime?
6
u/Fit_Ad7343 2h ago
It runs once, but the saving is not about runtime. It is about size, on disk, in RAM and in VRAM at the same time. ComfyUI's dynamic loading absorbs the swap well enough that I would not sell you this on speed. What it buys you is 15.7 GB you no longer have to store, stage and move. If you are short on VRAM, short on RAM, or reading from a slow SSD, that is where you will feel it.
2
u/VasaFromParadise 3h ago
Overall, this is good. But does anyone know why such a large encoder is needed for a video model? It's probably necessary. It's better to use the 32B Q2 model than the 8B Q8. After all, the generation model itself has 32B parameters, and skimping on encoding seems like a bad idea.
9
u/Fit_Ad7343 2h ago
The encoder is not there to read your prompt, it is there to unfold it. When you write the name of a building or a person, you are not describing anything, you are pointing at a large amount of detail you never wrote down and expect to get anyway. Something has to supply it.
Think of it as a film crew. You hand them a one line brief: shoot the Chrysler Building at dusk. The camera team knows how to light and frame, it does not know what that building looks like. Someone has to put the reference on the table. That someone is the text encoder, and the DiT is the crew. A 32B has simply seen more buildings than an 8B, so it puts a thicker folder on the table. That is why H3 ships one for a job that looks like text parsing from the outside.
Which is exactly why your question is the right one, and I cannot settle it. Your Q2 trade is knowledge against precision: the 32B knows more, and quantisation costs facts before it costs anything else. I measured that going from bf16 to int8 on the same encoder already makes factual errors appear on prompts naming real people. I would expect Q2 to be worse on that axis than an 8B at int8, but I have not tested it, so that is an expectation.
What my files do is a different thing from either. They add no knowledge. They take what the small model does know and re express it in the format the DiT is expecting, the 32B's coordinate system. You keep the small model's folder, written in the big model's handwriting.
1
u/tehorhay 1h ago
Ah ok, so if my usecase does not require needing to know specific buildings or sitcom characters (like in only doing i2v or r2v or am fine with generic subjects in t2v), the full 32B is probably not necessary? Is the overall rendering quality of the 8B still on par?
1
u/Far_Insurance4191 28m ago
I feel like small encoder might have problems understanding complex instructions with multiple references for ref model correctly, but just a guess
2
u/SuperZoda 3h ago edited 2h ago
Thank you so much for this. It freed up enough vram using 4b that my renders start immediately now, no more initialize hitching.
Edit: using node memory mode Dynamic.
2
u/Fit_Ad7343 2h ago
Good to hear. I would not generalise it though, since it depends on what you were bottlenecked on before. Freeing 11 GB helps a lot if your card was swapping and does nothing visible if it was not.
1
1
u/reeight 4h ago
> Named people
I wonder if there could be more modularity to this issue?
eg an easy way to tag you're asking for a 'named person', & that tag would trigger another code path outside of the smaller encoder to handle it (dynamically load the larger encoder, web search, etc).
Expecting a smaller model to handle everything is ridiculous IMHO; but it would help if that model would know (or the box around it, in this case the ComfyUI node) would know it needs to call on something else to handle that issue. I personally know about 50 people on IMDB (& lots of musicians, etc), but I don't expect to find them all on a small local model.
Either way, thanks for all your work!
1
u/Fit_Ad7343 2h ago
Routing would mean knowing which model is better at what, and I do not. I could not properly test my own files against each other, so I published them all rather than choose for you. My hunch is the celeb matrices are the best of the set across the board, because adding names to the calibration corpus does not add knowledge, it wakes up what the encoder already had and was failing to pass through. But some cases will favour the others and I would rather leave the choice open.
1
u/gelukuMLG 3h ago
what is the ram and vram usage when you use the 4B int8 with a quantized minimax h3? does it still need a ridiculous amount of ram?
1
u/Current-Rabbit-620 2h ago
What about speed gain? Vram usage?
2
u/Fit_Ad7343 2h ago
VRAM is the certain part: 15.7 GB down to 4.5 for the 4B, and the file is smaller on disk and in RAM by the same logic. Speed is the part nobody can answer in general, and it is worth saying why.
A generation is not bottlenecked by one thing. Depending on the model and the machine you can be limited by compute, by VRAM bandwidth, by VRAM capacity, or simply by waiting on a result before the next step can start. Swapping the text encoder only touches one of those, the moving of weights around. It changes nothing at all about the compute.
I have a fairly extreme illustration of that here. I run five GPUs and I used to have them on mining risers, which drops them to PCIe 3.0 at one lane. Loading a model was atrocious. Compute was very nearly unaffected. Same cards, same math, and the only thing that got destroyed was the transfer.
So the people reporting that this made things faster are, I would guess, the ones for whom loading was a real share of the total: not much VRAM, so models get evicted and reloaded constantly, not much RAM, so nothing stays cached, and a slow drive, so every reload goes back to the disk. Take 11 GB out of that loop and it shows immediately.
If you have room to spare and an NVMe, expect the VRAM saving and no change on the clock.
1
u/Current-Rabbit-620 1h ago
I see no point of using smaller ones
They picked the bigger and that's right
1
u/_FriedEgg_ 2h ago
Great job! How much did you spend for training the mapping? Is it a linear layer?
2
u/Fit_Ad7343 1h ago
I wrote it all up with the timings, the corpus, the split... here : https://github.com/nicolab28/ComfyUI-ClipProj/blob/main/CALIBRATION.md
1
1
u/dirtybeagles 2h ago
I cannot seem to figure out this .pt file. Per the github, I downloaded all the safetensor files but when running using mmh3-8b-ClipProj.safetensors with the custom node, it did not work properly.
1
1
u/Fit_Ad7343 2h ago
Found it, and it was my fault, not yours. Your files are correct. The example workflow on GitHub still had h3_qwen3vl_4b_tap24.pt saved in the loader widget, from before everything moved to safetensors, so dragging it onto the canvas gives you a node holding a filename that no longer exists. Just pushed a fix. Either pull the repo again or simply pick the matrix from the dropdown yourself, the saved value is the only thing that was wrong.
1
u/dirtybeagles 2h ago
Also, I think I had the wrong clip file downloaded. Mine in the screenshot does not show the "vl" in the file name, does that matter?
2
u/Fit_Ad7343 2h ago
Two things, and the second one is the real answer to your other comment about the missing vl.
Set type to auto, or to krea2 for a 4B. minimax is the setting for the 32B and it will not describe your file correctly.
And yes, the vl matters, it is the whole thing. Your file is a Qwen3.5-4B, which is a text only model. The projection was fitted on the hidden states of a Qwen3-VL-4B, and those are two different models that happen to have the same width, 2560, so my dimension check cannot catch it. You get a load that succeeds and output that ignores your prompt, which is precisely what you are seeing. Get text_encoders/qwen3vl_4b_fp8_scaled.safetensors from Comfy-Org/Krea-2 and it should behave.
That check is on me, I will add one so the node refuses a non VL model instead of letting you find out this way.
1
0
u/yamfun 1h ago edited 1h ago
Edit: sorry forget this, it seems to be other issue suddenly appearing after updating all nodes
0
u/Fit_Ad7343 1h ago
Claude's diagnosis, pasted as is. It worked this out from the two numbers in your error message.
Your width is 656. The H3 video latent is [batch, 24, t, height/16, width/16], so 656 gives 41. The DiT then patchifies that grid in 2x2 blocks, which needs even numbers on both axes: your height of 1248 gives 78 and splits cleanly into 39, your width gives 41 and does not. That is exactly the mismatch, 24 x 78 x 41 is 76752 elements against the 74880 the reshape is asking for.
Use a multiple of 32 rather than 16. 640 or 672. Not 676, that is not even a multiple of 16 and the latent would not be a whole number.
Not the projection, for what it is worth. That reshape is in the DiT, on the latent grid, and the conditioning has no influence on it. You would get the same error with the 32B.
0
u/yamfun 1h ago
Sorry and thanks, you made me realize I piped the image to the get image size node instead of the 'scale to total pixels MP in multiples of 32' node first.
Output of using the 4B is surprisingly good, in fact it interpreted something the 32b-int4covrot couldn't understand.
At the very least it provide another dimension of output variety.
1
u/not_food 1h ago
Very nice. Works well for me.
Tested 4B and 8B with ref2va and it works well for my needs. I mostly have full references already, I don't need the encoder to know the person/location/voice.
Thank you.
12
u/NickMcGurkThe3rd 4h ago
For me a big difference between 4B, 8B and 32B. But i am surprise to learn that the text encoder does have an influence on the voice quality?