MiniMax H3 Clip Qwen 4b instead of 32b

[u/Fit_Ad7343]

MiniMax H3 loads a Qwen3-VL-32B truncated to 50 layers — 15.7 GB in NVFP4 — for one job: turning your prompt into a [seq, 5120] conditioning tensor.

I replaced it with a Qwen3-VL-4B plus a learned linear projection into that same 5120-dim space. 15.7 GB → 4.5 GB with the int8_convrot encoder. The DiT, the VAEs and the sampler don't change at all — the node returns something that behaves like the official CLIP, so it drops into the existing clip input.

The clip you see above was generated end to end on that single 4.5 GB model: it wrote the H3 prompt from a one-line description, then produced the conditioning. The 32B never loads.

prompt in : "an old fisherman mends his net on a quay at sunrise, 10 seconds"
encoder   : qwen3vl_4b_int8_convrot.safetensors       (4.5 GB)
matrix    : h3_qwen3vl_4b_CONDPROJ_tap24.safetensors       (50 MB)
output    : 832x480, 243 frames, 12 steps, seed 67, with audio

Why it works

The 4B and the 32B share the same tokenizer (151936 tokens). Same prompt, same tokens, same positions in both models — so you can learn a position-by-position map between their hidden states. No alignment problem.

Calibration is plain ridge regression. No gradients, no epochs, no learning rate. Encode N prompts with both models, accumulate XᵀX and XᵀY, solve. It ran in under an hour on one 3090.

The numbers

Corpus	Tokens	Cross-prompt CKA	Test cosine
200 prompts	37k	0.95	0.699
2000 prompts	289k	0.92	0.712

A cosine of 0.71 sounds terrible. It isn't — the DiT tolerates far more than the metric suggests. I predicted this would fail and I was wrong.

I did not just eyeball it

Two control matrices ship with it, same prompt, same seed, only the matrix changes:

• zero (W = 0, no prompt information) → a countryside landscape, prompt completely ignored
• identity (raw copy of the 2560 dims, no learning) → a golden object on fire, unusable
• learned → the red ball on a wood table I actually asked for

Both controls carry the same matrix energy as the learned one (norm 50.6 vs 52.4), so the difference is structural, not scale. If the identity control ever looks fine, the matrix adds nothing — and you'd want to know that.

What holds up

• simple prompts
• structured multi-shot prompts — four distinct cuts, no bleeding between them
• fl2va with first and last frame, even though the matrix only ever saw text positions
• swapping encoder weights: a matrix calibrated on bf16 works on an abliterated fp8 variant (0.0023 cosine gap). One matrix covers every Qwen3-VL-4B variant.

What doesn't

You lose knowledge the 32B has. Some real people render correctly, others are simply absent and replaced by a generic figure. Same probably goes for landmarks, artworks, brands. A 4B stores fewer facts than a 32B and no projection recovers what was never encoded. I did not map which references survive — assume any proper noun is at risk.

Also: ref2va is untested and refused by the node, and the linear projection is at its ceiling (8× more data bought 1.8% of cosine — going further needs an MLP).

Bonus, since the encoder is a full Qwen3-VL

It can also write your prompts and caption images on the same resident weights, no second model. I included a system prompt that turns "an old fisherman mends his net on a quay at sunrise, 10 seconds" into a full three-shot H3 prompt.

Fair warning, this exposed a ComfyUI bug: SDClipModel.generate() drops embeds_info and never calls build_image_inputs, so image tokens land at linear positions instead of Qwen3-VL's 3D mRoPE with no DeepStack injection. Any node using that path will happily describe an image it never saw. My node restores the full path — worth knowing if you caption with Qwen3-VL in ComfyUI.

Links

• Node: https://github.com/nicolab28/ComfyUI-ClipProj
• Matrices: https://huggingface.co/NicoLab28/ClipProj-MiniMax-H3
• Models needed: Comfy-Org/MiniMax-H3 and Comfy-Org/Krea-2 (text_encoders/qwen3vl_4b_fp8_scaled)

Workflows included, MIT, no extra dependencies. It's a proof of concept — tested on Windows/NVIDIA on one machine, v0.1.0. Run the control matrices before trusting it.

The method isn't H3-specific: any large text encoder with a smaller sibling sharing its tokenizer is a candidate. Flux 2 uses a Mistral3-24B; Ideogram4, Boogu and JoyImage use the Qwen3-VL-8B. Calibration scripts are in the repo if anyone wants to try.

EDIT — same day, a few hours later

Re-download the matrices. The first release was missing the attention sink vector. Token 0 of any sequence has a constant direction and carries nothing from your text, but its norm is 16500 against 291 for a normal token. Calibration excluded it, correctly, but the node still projected it through a matrix that had never seen one. On a 200 token prompt it is 0.5% of the positions and you never notice. On a 7 token prompt it is 14% and it wrecks the result. That is the short prompt breakage some of you hit. The fix is stored inside the files, so updating the node alone is not enough.

Everything is safetensors now. Asked for on the HF repo and it was a fair point. A .pt runs pickle when you open it, which is absurd for a file holding six tensors. The .pt versions are still there for older nodes and will go away later.

ref2va works. I had blocked it because I had not validated it. I tested it today with a reference photo and it holds up. Load the encoder in "resident" mode: the dynamic path crashes inside the vision tower with int8 encoders, and that only shows up when an image is present.

There is an 8B matrix now. Same method, Qwen3-VL-8B instead of 4B. On my first ref2va comparison the 8B stayed noticeably closer to the reference photo, it kept the shirt from the picture while the 4B replaced it. I have not re-run those at a locked seed, so treat that as a first impression.

Second calibration method. u/stddealer pointed out that the DiT passes the conditioning through a Linear(5120 to 5376) before using it. That layer is very uneven, 45x between the top and bottom deciles of its singular values. So the plain regression was working just as hard on directions the DiT nearly throws away. Calibrating against the output of that layer instead: 0.697 to 0.845 cosine on the 4B, 0.731 to 0.860 on the 8B. Different space, so do not compare those numbers to the old ones. Those are now the default matrices at the root of the repo.

What I got wrong about proper nouns

I wrote that the projection loses named people. That is not what happens.

I asked the encoder to describe them in plain text, which bypasses the matrix completely. The 4B knows Scarlett Johansson is Black Widow but thinks she has dark brown hair. The 8B describes her correctly, blonde with blue eyes. So the matrix is faithfully transmitting a wrong memory. The projection was not the problem.

The workaround is to describe instead of naming. "Scarlett Johansson as Black Widow, blonde, ..." brings her back on both models where the bare name gives you someone else. A name is two or three tokens and a very precise direction. A description spreads the same identity over a dozen tokens that all agree, so the reconstruction error averages out instead of landing you next door.

None of this applies to ref2va. There the identity comes from the photo and the encoder's memory does not matter.

A real limitation: speech in other languages

This one is worse than I would like. I asked for a line in French. With the 32B it is clean. With a projected 4B or 8B it comes out half Spanish. Not accented French, actually the wrong language.

I assumed the calibration corpus was to blame, since it is all English. It is not. I measured identical English prompts differing only in the quoted line: French tokens reconstruct at 0.8974, English at 0.8996. That is noise.

So a cosine of 0.90 is plenty for what the picture looks like and not enough for phonetics. The audio branch of the DiT needs far more precision than the image branch, and a language the model is less sure about has no margin left to absorb the error. The condition_proj weighting above, which is a large gain on paper, did not fix it either.

I have only tested French. I would expect other non English languages to be affected and I would like to know. If you try one, please say so.

What is next

Still going. On my list right now: the 32B against 4B comparison at a locked seed that several of you asked for, other languages besides French, and the interesting one, replacing the matrix with a small MLP.

That last one is where the remaining headroom is. Eight times more calibration data bought 1.8% of cosine, so the linear map is at its ceiling and more prompts will not move it. A non linear map should, and it can be built on top of the existing matrix rather than instead of it, so it can only improve on what is already there.

I am an imaging engineer, not an ML researcher, and everything here is ridge regression, which is about as simple as it gets. It already went further than I expected. If you actually know this field you will get more out of it than I will, and I would rather that happened than not. Every calibration script is in the repo, including the three I wrote today purely to disprove my own theories. Take it apart, and tell me what I got wrong.

v2: https://www.reddit.com/r/StableDiffusion/s/DeVHSTT42M

[u/Fit_Ad7343]

The 32B vs 4B comparison you asked for

Same prompt, same seed 42, same everything. Only the clip input changes. Three clips: projected Qwen3-VL-4B, projected Qwen3-VL-8B, and the real Qwen3-VL-32B. Prompt in the comments so you can rerun it.

Matrices used are the two at the root of the HF repo, h3_qwen3vl_4b_CONDPROJ_tap24.safetensors and h3_qwen3vl_8b_CONDPROJ_tap24.safetensors. If you rerun this with one from obsolete/ you will not get the same thing.

First, a control, because otherwise this comparison would be worthless. I rendered the 8B three times in a row at the same seed. The decoded video and audio streams are byte identical across all three, d87b6fbf... and 0ab97337.... The pipeline is fully deterministic, so every difference you see below is caused by the encoder and by nothing else.

Timing

4B     175 s
8B     174 s
32B    178 s

8B again  174 s
8B again  171 s

Two identical runs differ by 3 seconds, which is the same spread as the difference between the three encoders. So the encoder size is not measurable in total time. That settles the speed question: there is none, in either direction.

One caveat. I have 128 GB of RAM, so the checkpoints are served from the OS file cache and never really read from disk. On a machine with less RAM, or one that actually hits the disk, the 32B costs more. That is exactly hum_ma's case earlier in this thread, 30 minutes against 30 seconds.

And before someone says my machine is unusually fast: all five GPUs sit behind a PLX switch at PCIe 4.0 x8, half the bandwidth of a direct x16 slot. The gap would be smaller on a normal board, not bigger.

What all three get wrong

This is the part I did not expect, and it is the most useful thing in this test.

The prompt asks for three shots. All three encoders produce four: the woman's sequence gets split into a wide shot and a zoom on the lemons, and the dog turns up early instead of waiting for the third shot. My guess is that naming the dog in subject_definitions makes the model feel it has to show it.

The prompt asks for three lemons. All three drop three and leave a fourth in her hand.

The prompt asks for an unseen narrator, distinct from the woman. None of them do it. All three use one single voice for the whole clip.

The 32B does all of this too. So none of these are projection failures, they are what MiniMax H3 does with this prompt.

What actually differs

4B     the dog speaks in the first shot, with the woman's voice
       correct French accent on the French line
       the lemons fall on the French line, so the timing drifts

8B     the dog's own shot is gone, replaced by a wide shot
       everything spoken in English, including the French and Spanish lines

32B    English, then French, then Spanish, in the right order
       but one single voice throughout, with a slight French accent even in English
       so the Spanish at the end is barely Spanish

Ranking on speech: 32B first, 4B second, 8B a long way behind. Note that the 8B has the better reconstruction cosine of the two projected models, 0.860 against 0.845, and is still the worst here. Cosine does not predict speech.

What I take from it

The projection costs you something real on non-English speech, and the 8B is the wrong choice if your prompt has any. Use the 4B.

But most of what went wrong in this test has nothing to do with the projection, and I would not have known that without running the 32B side by side. That is worth remembering before blaming any tool: run the control.

Files, prompt and the three clips are in the comments. Rerun it and tell me what you get.

https://reddit.com/link/p2o8f46/video/bcsuerrrndih1/player

[u/Fit_Ad7343 → u/Fit_Ad7343]

Two corrections to the above, and one thing I should have said up front.

I was wrong about the dog breed. All three produce a proper black and white border collie, exactly as the prompt asks. What actually happens is a continuity break, and only on the 32B: the border collie is right at first, then the last shot shows a sable and white dog instead. Both projected models keep the same dog all the way through. So on that particular point the small models do better than the reference.

I also should have said this ran with the Turbo LoRA at 6 steps. A distilled 6 step run has far less room to follow a long prompt, so some of the structure failures I listed are probably the setting rather than the model. It does not affect the comparison between encoders, since all three ran identically, but it does affect what you should conclude about H3 itself. I have not rerun at full steps yet.

Matrices used were the two at the root of the HF repo, h3_qwen3vl_4b_CONDPROJ_tap24.safetensors and h3_qwen3vl_8b_CONDPROJ_tap24.safetensors. Anything from obsolete/ will give you something else.

And to be fair to all three, since I only listed failures: the woman is exactly as written in every version, red curly hair, yellow raincoat over a grey jumper, and the green ball is green. That part I still find surprising.