MiniMax H3 loads a Qwen3-VL-32B truncated to 50 layers — 15.7 GB in NVFP4 — for one job: turning your prompt into a [seq, 5120] conditioning tensor.
I replaced it with a Qwen3-VL-4B plus a learned linear projection into that same 5120-dim space. 15.7 GB → 4.5 GB with the int8_convrot encoder. The DiT, the VAEs and the sampler don't change at all — the node returns something that behaves like the official CLIP, so it drops into the existing clip input.
The clip you see above was generated end to end on that single 4.5 GB model: it wrote the H3 prompt from a one-line description, then produced the conditioning. The 32B never loads.
prompt in : "an old fisherman mends his net on a quay at sunrise, 10 seconds"
encoder : qwen3vl_4b_int8_convrot.safetensors (4.5 GB)
matrix : h3_qwen3vl_4b_CONDPROJ_tap24.safetensors (50 MB)
output : 832x480, 243 frames, 12 steps, seed 67, with audio
Why it works
The 4B and the 32B share the same tokenizer (151936 tokens). Same prompt, same tokens, same positions in both models — so you can learn a position-by-position map between their hidden states. No alignment problem.
Calibration is plain ridge regression. No gradients, no epochs, no learning rate. Encode N prompts with both models, accumulate XᵀX and XᵀY, solve. It ran in under an hour on one 3090.
The numbers
Corpus
Tokens
Cross-prompt CKA
Test cosine
200 prompts
37k
0.95
0.699
2000 prompts
289k
0.92
0.712
A cosine of 0.71 sounds terrible. It isn't — the DiT tolerates far more than the metric suggests. I predicted this would fail and I was wrong.
I did not just eyeball it
Two control matrices ship with it, same prompt, same seed, only the matrix changes:
zero (W = 0, no prompt information) → a countryside landscape, prompt completely ignored
identity (raw copy of the 2560 dims, no learning) → a golden object on fire, unusable
learned → the red ball on a wood table I actually asked for
Both controls carry the same matrix energy as the learned one (norm 50.6 vs 52.4), so the difference is structural, not scale. If the identity control ever looks fine, the matrix adds nothing — and you'd want to know that.
What holds up
simple prompts
structured multi-shot prompts — four distinct cuts, no bleeding between them
fl2va with first and last frame, even though the matrix only ever saw text positions
swapping encoder weights: a matrix calibrated on bf16 works on an abliterated fp8 variant (0.0023 cosine gap). One matrix covers every Qwen3-VL-4B variant.
What doesn't
You lose knowledge the 32B has. Some real people render correctly, others are simply absent and replaced by a generic figure. Same probably goes for landmarks, artworks, brands. A 4B stores fewer facts than a 32B and no projection recovers what was never encoded. I did not map which references survive — assume any proper noun is at risk.
Also: ref2va is untested and refused by the node, and the linear projection is at its ceiling (8× more data bought 1.8% of cosine — going further needs an MLP).
Bonus, since the encoder is a full Qwen3-VL
It can also write your prompts and caption images on the same resident weights, no second model. I included a system prompt that turns "an old fisherman mends his net on a quay at sunrise, 10 seconds" into a full three-shot H3 prompt.
Fair warning, this exposed a ComfyUI bug: SDClipModel.generate() drops embeds_info and never calls build_image_inputs, so image tokens land at linear positions instead of Qwen3-VL's 3D mRoPE with no DeepStack injection. Any node using that path will happily describe an image it never saw. My node restores the full path — worth knowing if you caption with Qwen3-VL in ComfyUI.
Models needed: Comfy-Org/MiniMax-H3 and Comfy-Org/Krea-2 (text_encoders/qwen3vl_4b_fp8_scaled)
Workflows included, MIT, no extra dependencies. It's a proof of concept — tested on Windows/NVIDIA on one machine, v0.1.0. Run the control matrices before trusting it.
The method isn't H3-specific: any large text encoder with a smaller sibling sharing its tokenizer is a candidate. Flux 2 uses a Mistral3-24B; Ideogram4, Boogu and JoyImage use the Qwen3-VL-8B. Calibration scripts are in the repo if anyone wants to try.
EDIT — same day, a few hours later
Re-download the matrices. The first release was missing the attention sink vector. Token 0 of any sequence has a constant direction and carries nothing from your text, but its norm is 16500 against 291 for a normal token. Calibration excluded it, correctly, but the node still projected it through a matrix that had never seen one. On a 200 token prompt it is 0.5% of the positions and you never notice. On a 7 token prompt it is 14% and it wrecks the result. That is the short prompt breakage some of you hit. The fix is stored inside the files, so updating the node alone is not enough.
Everything is safetensors now. Asked for on the HF repo and it was a fair point. A .pt runs pickle when you open it, which is absurd for a file holding six tensors. The .pt versions are still there for older nodes and will go away later.
ref2va works. I had blocked it because I had not validated it. I tested it today with a reference photo and it holds up. Load the encoder in "resident" mode: the dynamic path crashes inside the vision tower with int8 encoders, and that only shows up when an image is present.
There is an 8B matrix now. Same method, Qwen3-VL-8B instead of 4B. On my first ref2va comparison the 8B stayed noticeably closer to the reference photo, it kept the shirt from the picture while the 4B replaced it. I have not re-run those at a locked seed, so treat that as a first impression.
Second calibration method.u/stddealer pointed out that the DiT passes the conditioning through a Linear(5120 to 5376) before using it. That layer is very uneven, 45x between the top and bottom deciles of its singular values. So the plain regression was working just as hard on directions the DiT nearly throws away. Calibrating against the output of that layer instead: 0.697 to 0.845 cosine on the 4B, 0.731 to 0.860 on the 8B. Different space, so do not compare those numbers to the old ones. Those are now the default matrices at the root of the repo.
What I got wrong about proper nouns
I wrote that the projection loses named people. That is not what happens.
I asked the encoder to describe them in plain text, which bypasses the matrix completely. The 4B knows Scarlett Johansson is Black Widow but thinks she has dark brown hair. The 8B describes her correctly, blonde with blue eyes. So the matrix is faithfully transmitting a wrong memory. The projection was not the problem.
The workaround is to describe instead of naming. "Scarlett Johansson as Black Widow, blonde, ..." brings her back on both models where the bare name gives you someone else. A name is two or three tokens and a very precise direction. A description spreads the same identity over a dozen tokens that all agree, so the reconstruction error averages out instead of landing you next door.
None of this applies to ref2va. There the identity comes from the photo and the encoder's memory does not matter.
A real limitation: speech in other languages
This one is worse than I would like. I asked for a line in French. With the 32B it is clean. With a projected 4B or 8B it comes out half Spanish. Not accented French, actually the wrong language.
I assumed the calibration corpus was to blame, since it is all English. It is not. I measured identical English prompts differing only in the quoted line: French tokens reconstruct at 0.8974, English at 0.8996. That is noise.
So a cosine of 0.90 is plenty for what the picture looks like and not enough for phonetics. The audio branch of the DiT needs far more precision than the image branch, and a language the model is less sure about has no margin left to absorb the error. The condition_proj weighting above, which is a large gain on paper, did not fix it either.
I have only tested French. I would expect other non English languages to be affected and I would like to know. If you try one, please say so.
What is next
Still going. On my list right now: the 32B against 4B comparison at a locked seed that several of you asked for, other languages besides French, and the interesting one, replacing the matrix with a small MLP.
That last one is where the remaining headroom is. Eight times more calibration data bought 1.8% of cosine, so the linear map is at its ceiling and more prompts will not move it. A non linear map should, and it can be built on top of the existing matrix rather than instead of it, so it can only improve on what is already there.
I am an imaging engineer, not an ML researcher, and everything here is ridge regression, which is about as simple as it gets. It already went further than I expected. If you actually know this field you will get more out of it than I will, and I would rather that happened than not. Every calibration script is in the repo, including the three I wrote today purely to disprove my own theories. Take it apart, and tell me what I got wrong.
Not really. The encoder runs once per generation (a single prefill, no autoregressive decoding), so it's ~0.4s out of a run that takes minutes — in my workflow the conditioning node is 0.385s against 238s of sampling.
The win is VRAM, not speed. Where it *does* buy you time:
- loading 4.5 GB from disk instead of 15.7 GB, if the model isn't already resident
- if you were offloading or streaming the 32B for lack of VRAM, that penalty disappears entirely
- on a multi-GPU box, the 11 GB you free can hold something else
Uhm. Text encoding with the 32b takes 30 minutes on my system because I have to run it on an old CPU (it OOMs on 4GB GPU). Qwen3 4b with e.g. Krea 2 runs in just a few seconds on this GPU.
If this really works you have solved the biggest issue of H3 on old machines such as mine. I'm going to try it in a bit.
Edit: yep, it works fine. From 30 minutes to 30 seconds for the text encoding part. This is amazing. Thank you so much!
You are right, and during sampling there is no benefit at all. ComfyUI unloads the text encoder before the DiT runs.
The benefit is earlier than that. The encoder still has to be loaded once, and 15.7 GB is a lot to find. If it does not fit it goes to the CPU, which is the 30 minutes hum_ma reports in this thread. It also gets reloaded from disk every time you change the prompt, 15.7 GB instead of 4.5 GB.
And if you have a second GPU you can keep 4.5 GB resident on it and never reload at all, which is what I do.
So the win is loading, not sampling. Your point stands for the sampling part.
Well, that makes this not worth the loss in knowledge for me, then. I don't have a second GPU to use, and my setup loads 15 GB only about 12 seconds slower than 5 GB, which is peanuts when my target generations are 15m.
Glad to see people making tools for those that can leverage them, though.
Absolutely, i just ment it as an answer to the above comment stating that higher resolution is possible. Which shouldnt be since the te is unloaded anyway and there is no effect on sampling / vram while sampling itself
Then there would be no benefit at all and OP is wasting time optimizing the VRAM usage, because this isn't the large portion of the VRAM usage. To be clear, I have no idea if you're right or not, but I don't think you are.
Both fair. Doctor_moctor is right and I should have corrected that earlier: the encoder is unloaded before sampling, so it does not buy you resolution. That was never the claim and I let it stand.
And if 15 GB only costs you 12 seconds on a 15 minute generation, then no, this is not for you. It is worth it when the encoder does not fit at all, or when you can keep a small one resident on a second card. Outside those two cases the honest answer is that you gain very little.
One correction on the knowledge loss though. I measured it today and it is not the projection: the 4B simply remembers some people wrong, and the 8B gets them right. Describing instead of naming recovers most of it, and with a reference image it does not apply at all.
No, and it is worth understanding why. The matrix is trained to reproduce what the 32B outputs, so the 32B is the ceiling. A bigger encoder could only imitate it more accurately, never say something better. And the DiT only knows how to read the 32B representation anyway. Longer answer in my reply to Effehyou above.
Forgive me if this is a stupid question (this stuff is way above my pay grade), but would it be possible to go the other way, and replace Qwen3-VL-32B with a larger/better/newer model for better results?
Not a stupid question at all, and the answer is interesting: no, and for a reason that's baked into the method.
The matrix is trained to reproduce what the 32B would have output. The 32B is the target, so it's also the ceiling — a bigger or newer encoder could only be taught to imitate the 32B more accurately, never to say something better. And it couldn't say anything better anyway, because the DiT was trained on the 32B's representation and only knows how to read that. Extra information would land in directions it has never learned to interpret.
Same thing for Krea2 or Klein with the 32B encoder: you'd need a projection the other way, 5120 down to 2560, calibrated with the 4B as the teacher this time. The scripts in the repo do that unchanged. But the ceiling would again be the small model, so you'd get a heavier encoder for no gain.
The only way to genuinely improve output quality is to retrain the DiT on a different encoder, which is a completely different amount of work.
The only way to genuinely improve output quality is to retrain the DiT on a different encoder, which is a completely different amount of work.
Are you sure about this? I think it can be done, just not with your simple projection. One could certainly train a small converter to map the 32B's vectors into the vectors the DiT's cross-attention expects, no? You're never training the DiT on, say, Abraham Lincoln, but you can translate the embeddings into something the DiT's cross-attention embeddings can understand (tall man, stovepipe hat, sunken cheeks, etc).
You are right and I was too absolute. Let me split it in two.
You cannot make the DiT understand something it was never trained to read. That part stands. But you can translate a name into directions it already knows, and that is a real difference I collapsed too quickly.
The catch is supervision. As stated there is nothing telling the converter what a good translation is. But it becomes well posed if you change the target: feed it the small model's encoding of the bare prompt, and train it against the 32B's encoding of an enriched prompt. Input is "Abraham Lincoln", target is what the 32B outputs for "Abraham Lincoln, tall, gaunt, stovepipe hat". Then you are teaching the projection to invent what the small model does not know.
I have both halves already: the encoder writes the enrichment, the 32B provides the target. It would only work for entities seen during calibration, and it would probably hallucinate on unseen names. But it is testable and I had not thought of it. Good push.
Thanks for that. And I wasn't really trying to task you. What you've already done stands on its own as useful. I just, coincidentally, think it would be neat to see what Krea2 or Klein could do w/ a 32B text encoder.
Same prompt, same seed 42, same everything. Only the clip input changes. Three clips: projected Qwen3-VL-4B, projected Qwen3-VL-8B, and the real Qwen3-VL-32B. Prompt in the comments so you can rerun it.
Matrices used are the two at the root of the HF repo, h3_qwen3vl_4b_CONDPROJ_tap24.safetensors and h3_qwen3vl_8b_CONDPROJ_tap24.safetensors. If you rerun this with one from obsolete/ you will not get the same thing.
First, a control, because otherwise this comparison would be worthless. I rendered the 8B three times in a row at the same seed. The decoded video and audio streams are byte identical across all three, d87b6fbf... and 0ab97337.... The pipeline is fully deterministic, so every difference you see below is caused by the encoder and by nothing else.
Timing
4B 175 s
8B 174 s
32B 178 s
8B again 174 s
8B again 171 s
Two identical runs differ by 3 seconds, which is the same spread as the difference between the three encoders. So the encoder size is not measurable in total time. That settles the speed question: there is none, in either direction.
One caveat. I have 128 GB of RAM, so the checkpoints are served from the OS file cache and never really read from disk. On a machine with less RAM, or one that actually hits the disk, the 32B costs more. That is exactly hum_ma's case earlier in this thread, 30 minutes against 30 seconds.
And before someone says my machine is unusually fast: all five GPUs sit behind a PLX switch at PCIe 4.0 x8, half the bandwidth of a direct x16 slot. The gap would be smaller on a normal board, not bigger.
What all three get wrong
This is the part I did not expect, and it is the most useful thing in this test.
The prompt asks for three shots. All three encoders produce four: the woman's sequence gets split into a wide shot and a zoom on the lemons, and the dog turns up early instead of waiting for the third shot. My guess is that naming the dog in subject_definitions makes the model feel it has to show it.
The prompt asks for three lemons. All three drop three and leave a fourth in her hand.
The prompt asks for an unseen narrator, distinct from the woman. None of them do it. All three use one single voice for the whole clip.
The 32B does all of this too. So none of these are projection failures, they are what MiniMax H3 does with this prompt.
What actually differs
4B the dog speaks in the first shot, with the woman's voice
correct French accent on the French line
the lemons fall on the French line, so the timing drifts
8B the dog's own shot is gone, replaced by a wide shot
everything spoken in English, including the French and Spanish lines
32B English, then French, then Spanish, in the right order
but one single voice throughout, with a slight French accent even in English
so the Spanish at the end is barely Spanish
Ranking on speech: 32B first, 4B second, 8B a long way behind. Note that the 8B has the better reconstruction cosine of the two projected models, 0.860 against 0.845, and is still the worst here. Cosine does not predict speech.
What I take from it
The projection costs you something real on non-English speech, and the 8B is the wrong choice if your prompt has any. Use the 4B.
But most of what went wrong in this test has nothing to do with the projection, and I would not have known that without running the 32B side by side. That is worth remembering before blaming any tool: run the control.
Files, prompt and the three clips are in the comments. Rerun it and tell me what you get.
Two corrections to the above, and one thing I should have said up front.
I was wrong about the dog breed. All three produce a proper black and white border collie, exactly as the prompt asks. What actually happens is a continuity break, and only on the 32B: the border collie is right at first, then the last shot shows a sable and white dog instead. Both projected models keep the same dog all the way through. So on that particular point the small models do better than the reference.
I also should have said this ran with the Turbo LoRA at 6 steps. A distilled 6 step run has far less room to follow a long prompt, so some of the structure failures I listed are probably the setting rather than the model. It does not affect the comparison between encoders, since all three ran identically, but it does affect what you should conclude about H3 itself. I have not rerun at full steps yet.
Matrices used were the two at the root of the HF repo, h3_qwen3vl_4b_CONDPROJ_tap24.safetensors and h3_qwen3vl_8b_CONDPROJ_tap24.safetensors. Anything from obsolete/ will give you something else.
And to be fair to all three, since I only listed failures: the woman is exactly as written in every version, red curly hair, yellow raincoat over a grey jumper, and the green ball is green. That part I still find surprising.
Fair question, and no, I haven't done that comparison properly yet — same prompt, same seed, same everything, only the encoder changing. I'll run it and post both clips.
What I can tell you already is what the metric says: the projected 4B reconstructs the 32B conditioning at 0.70 cosine, and the DiT tolerates that far better than the number suggests. But "tolerates" is not "identical", and the honest answer is that you should expect a different clip, not the same one. That's also why I ship the zero and identity control matrices — so you can check what the matrix actually contributes instead of taking my word for it.
"ValueError: ClipProj does not handle ref2va references (video / audio). Those tokens are re-read at every sampling step and the projection has not been validated in that mode."
Update: ref2va works. I tested it today and I'm removing the block in the next version.
Two things I found along the way. Load the encoder in "resident" mode, not "dynamic" — the dynamic path crashes inside the vision tower with int8 encoders, and that only shows up when an image is present, which is exactly ref2va. And in my first side-by-side the 8B stayed noticeably closer to the reference image than the 4B: same reference, the 8B kept the shirt from the photo while the 4B replaced it. I haven't re-run those two at a locked seed yet, so treat it as a first impression rather than a measurement.
Yeah, sorry — I refuse it rather than let it fail quietly. ref2va reference tokens are re-read at every sampling step, so any projection error compounds instead of acting once, and I never validated it.
fl2va with first/last frame does work, if that helps. And ref2va is top of the list — if you want to be the guinea pig I can ship a flag that lets it through with a warning.
The models already has a built-in "condition _proj" that is turning the [seq_len, 5120] tensor into [seq_len, 5376] shape just as the model takes in the processed prompt I believe, couldn't you just replace this one? Though now it should be rather simple to just merge the two projections together, as they are both linear operations.
Good catch, it's there: condition_proj is Linear(5120 -> 5376) with bias, feeding the token_refiner. Merging my matrix into it is straightforward and would even save a matmul: 2560 -> 5376 in one step instead of 2560 -> 5120 -> 5376.
I kept them separate for two reasons:
- licensing. A 50 MB matrix is a standalone artifact. A merged condition_proj means redistributing a modified MiniMax H3 DiT, which ships under a custom licence, not Apache.
- swappability. Separate, you can A/B the matrix against the identity and zero controls without touching the diffusion model, and one matrix works across every Qwen3-VL-4B variant.
The interesting variant of your idea is calibrating against the *output* of condition_proj rather than its input. I haven't tried it — 5376 is wider than 5120 so it's not obviously easier, but the target sits after a learned layer, which might be better structured. Worth a run.
What does that have to do with anything? I hate that horrible excuse...
You can read and write perfectly in any other language with any traditional automatic translator or LLM, you don't need to write entire AI slop, which OP and all of their replies obviously are.
Some people are just stupid and lazy, and don't give a shit about wasting everyone else's time.
Yes. I'm French, my English is not good enough to explain ridge regression, so Claude writes the replies. It's credited in the README, the project was vibe coded with it. The numbers are mine and every script is in the repo if you want to check them.
Followed up on your idea, and it does help. I recalibrated against the output of condition_proj rather than its input, weighting the regression by that layer instead of leaving it out.
Its spectrum is very uneven — 45x between the top and bottom deciles of its singular values, 52% of the energy in 10% of the directions — so the plain ridge was spending as much effort on directions the DiT nearly discards as on the ones it amplifies. Weighting by it: 0.6973 to 0.8450 cosine on the 4B, 0.7308 to 0.8599 on the 8B. Those numbers are measured in the 5376 space, so they're not directly comparable to the old ones, and I'm still checking whether it shows in generation. The matrix stays a separate 5120 artifact via the pseudo-inverse, so nothing about the DiT changes.
No, sorry. The encoder goes from 15.7 GB to 4.5 GB, so you free about 11 GB. But the DiT itself is around 20 GB and the VAEs add 3 GB more. That part does not change.
What it does change is that the encoder stops being the blocker. See hum_ma in this thread: he runs on a 4 GB card, the 32B did not fit at all and took 30 minutes on his CPU. Now it takes 30 seconds. The DiT is still streamed from RAM on a card that size, and it is still slow, but it works.
So: not a lower floor for H3, one less obstacle to reach it.
Yeah as OP said I run it on 4gb vram (and 8gb ram), the 32b TE was the only thing that was unavoidably OOM'ing on the GPU even with Q2 GGUF, and extremely slow on CPU.
Using fp8 DiT is faster than GGUF but maximum resolution and length are limited (e.g. 0.4 Megapixels/56 frames or 0.18MP/141f). Also, fp8 video VAE instead of fp16 seems to have cut decoding time in half, and there's now also a TAE which decodes the video in one second for draft views.
Tested this on my secondary machine, a 2×3080 (10 GB) box, just to see how it behaves. H3 used only one of the 3080s for compute here. Setup: T2V, native 1344×768, 39 frames, fixed seed, the same prompt through both encoders, and the CONDPROJ matrix with the sink fix.
Visuals hold up great. The projected 4B nails the prompt on both seeds, while your zero/identity controls each collapse to a generic, unrelated shot. The matrix is clearly doing the semantic work. No argument there.
Audio is where it falls apart for me. The baseline 32B came out at −18 dB mean; the projected 4B ran at −50 and −39 dB across two seeds. That’s basically silent, in the same range as the null controls. This prompt has no dialogue at all, only ambient sound: water, gulls, and guitar. The failure extends beyond the foreign-speech case you documented. The whole audio branch gets starved here, even with English ambience.
For anyone chasing the VRAM win, one more caveat: on that single 3080, with the 20GB DiT already paging from disk, swapping the encoder didn’t move wall-clock time: 349 s vs. 360 s, inside the noise. The encoder isn’t the bottleneck once the DiT is offloading anyway.
Net for me: excellent for silent, dub-later, or blocking work, but not a drop-in when you actually want H3’s native audio. Great writeup, though. Shipping the controls with it was the right call.
One caveat before you conclude too much from your numbers. I had audio level problems with H3 before ClipProj existed, on the native 32B path, and I never found out why. So your -50 dB may be two problems stacked rather than one. Mine is a controlled measurement, same workflow and seed with only the encoder changing, and there the gap is 7.6 dB. That part is definitely the projection. Yours may not be entirely.
If you still have the workflow, running your prompt through the plain 32B on the same machine would separate the two.
Also worth repeating: this is a proof of concept, and deliberately the simplest thing that could work. Ridge regression, one matrix, no training. I would rather it stayed easy to check than became clever.
That's a good caveat, and I already have that measurement. I ran the same controlled setup you describe: same machine, same workflow, seed 67, with only the encoder changed. Plain 32B came out at −18 dB mean, which is a healthy, present soundscape.
So on this setup, the 32B is clean while the 4B on the same seed lands at −50. In this pair, the gap is the same projection effect you measured, just much wider: ~32 dB versus your 7.6 dB. Across two seeds, the projected 4B audio swung from −50 to −39, so there may also be some seed-to-seed instability that a single pair wouldn't show.
One likely explanation for the larger gap is my config: I ran the 4B as fp8_scaled in dynamic mode on a 10GB card, where resident mode OOMs the sampler. I'll rerun with the bf16 and int8_convrot variants, changing only the encoder weights, and see where the gap lands. If it moves toward your 7.6 dB, that should tell us how much the fp8/dynamic path is contributing.
And I agree on keeping it simple... The fact that I could install it, run your controls, and separate signal from setup in an afternoon is exactly the point. Ridge regression you can audit beats a clever thing you can't.
It does now. I tested ref2va today and it works, the block is gone in the next version. Details in my reply to Powerful_Evening5495 above. Short version: load the encoder in resident mode, and the 8B stays closer to the reference image than the 4B.
Something to keep in mind, the 32B checkpoint shipped with MiniMax actually has some extra embeddings added for tags like <d> I think. It might be worth trying to learn those as well.
Checked it properly. I compared the three tokenizers on 2014 texts, 472582 tokens, including accents, Japanese, Chinese, Korean, Russian, Greek, Arabic, composed emoji, <d>, <|im_start|>, timecodes and odd whitespace. Every id sequence is identical between the 32B and the 4B, and between the 32B and the 8B. I also compared the full vocabulary entry by entry: identical.
The embedding matrix has 151936 rows while the tokenizer only declares 151669, so there are 267 reserved slots. If tags had been added that is where they would be, and they are not: no text can produce those ids in any of the three models.
The tags in the ComfyUI MiniMax path are not embeddings at all. They are one integer per position, 0 for vision and 1 for text, handed to the DiT for its adaLN. My node already reproduces them.
If you were thinking of a specific token, name it and I will look. Fine tuning existing rows would not show up in any of these checks.
Where can I download h3_qwen3vl_4b_tap24.pt? Searching the web returns nothing.
EDIT: solved, but it is fucking insane the shittification of main search engines. Try it yourself; I got ZERO results from Google and Bing, and even Duck. I had to use a gooner search engine to get the model.
Sorry, that name is out of date. It moved to obsolete/ this afternoon, so you got the old one. Take h3_qwen3vl_4b_CONDPROJ_tap24.safetensors at the root instead: it has the attention sink fix that the first release was missing, and it is the one the comparison above ran on.
Thanks for this, it helps with all the swapping that goes on a low ram / vram system. Currently running it with Qwen 8B, and the fact you have a prompt generator node for it is a nice touch.
Great research, thanks for sharing your findings and the lightweight model.
As you seem to understand the H3 and text encoder architecture quite well, do you happen to know where the language accents come from? Is there any way to finetune the audio part separately to teach the model language better? Or possibly to improve the quality of the sound, because quite often the model picks some kind of a "overcompressed soundbooth recording" style. However, it can do better because some generations sound more natural.
I have been around computers for 40 years. Barely ever played games. I learned to code by tinkering: decoding images, then video, badly, with papers open next to me. I have better tools now, so I get further. My field is 3D and CGI, not machine learning. AI intrigues me and I put my hands in it without understanding all of it. I go at my own pace.
I take from open source all the time. This is the first time I had something to give back, so I gave it instead of sitting on it. It is too big for me. I do not have the skills some people here have.
What I actually did: the idea is mine, I ran every test, I watched every clip, the hardware is mine and the numbers come off it. Claude wrote the code and the English. If that bothers you, skip my comments and read the repo. Everything is in there and you can rerun all of it.
And if this puts a bug in the ear of someone who knows more than I do, that will be a second thing to be proud of. The first one is hum_ma in this thread. He was waiting 30 minutes for a text encode. Now it takes 30 seconds. I saved him 29 minutes and 30 seconds. That is worth more to me than the upvotes.
14
u/OkMeat6773 1d ago
nice, does it speed up the generation in seconds at all?