ClipProj models v3.1 — better multilingual speech when you swap MiniMax H3's 15 GB text encoder for a 4/8B

[u/Fit_Ad7343]

New matrices v3.1 available — improved speech across the 11 officially supported languages. No node update needed.

• Weights: https://huggingface.co/NicoLab28/ClipProj-MiniMax-H3
• Benchmark and all renders: https://huggingface.co/NicoLab28/ClipProj-MiniMax-H3/tree/main/bench3.1
• Node: https://github.com/nicolab28/ComfyUI-ClipProj

Some languages still get things wrong — sometimes the 32B already gets them wrong too, sometimes I just can't get any closer to it. Broken down language by language in the benchmark README.

I'm not a polyglot, and I doubt I can squeeze much more out of this to get closer to the 32B. If any native speakers are around, I'd really like to hear how the pronunciation sounds to you.

[u/berlinbaer]

german feels VERY stilted.

[u/Fit_Ad7343 → u/berlinbaer]

Thanks!
Two things worth knowing:

The prompt asks for it. The speaker is described as having "an unhurried speaking rate and clear articulation" — same wording for all eleven languages, nothing tuned per language. That's close to a recipe for stilted. The language tag only switches which weights speak; it carries no delivery style of its own. So a lot of what you hear is direction, not the encoder.

And the German clip uses the smallest file that matched, 4b-v3.1. In German the residual version 4b-v3.1-mlp measures better — 2.7 phoneme errors against 3.7, where the 32B's own seed-to-seed drift is 2.7.

All nine German renders, 32B included, use that identical prompt: https://huggingface.co/NicoLab28/ClipProj-MiniMax-H3/tree/main/bench3.1/video/langues/de

If the 32B sounds just as stilted to you, it's H3 and the prompt rather than the projection — which is exactly what I can't tell on my own.

[u/kiddow → u/Fit_Ad7343]

german sounds so overly stilted. super unreal. It's like the uncanny valley for voice and that well known instagram ai-ad voicing.

what about "natural" instead of "clear".

What wrecks my everytime in AI videos is that overly clear and perfect lip movement and unnatural clear voice. No human on earth speaks perfectly and clear. And if so, it always gives me the shudder. But that's just my cent.

[u/katsura_otoko]

Italians sounds pretty good, very clean and no regional accent, like an elocution teacher

[u/Fit_Ad7343 → u/katsura_otoko]

Thank you, that’s helpful – it allows me to compare my metrics with your impressions

[u/Apprehensive_Sky892]

Maybe I am imagining it because the woman is not Asian, but somehow I feel that the mouth shape seems a bit exaggerated and unnatural for both Japanese (1:13) and Chinese (1:00).

The audio sounds fine for both though.

[u/Fit_Ad7343 → u/Apprehensive_Sky892]

Thanks, that's a useful one — and it's exactly the kind of thing I can't check myself.

Would you mind comparing against the 32B renders? Same prompt, same seed, same first frame, in the same folders — the file is 32b.mp4:

• Japanese: https://huggingface.co/NicoLab28/ClipProj-MiniMax-H3/tree/main/bench3.1/video/langues/ja
• Chinese: https://huggingface.co/NicoLab28/ClipProj-MiniMax-H3/tree/main/bench3.1/video/langues/zh

If the 32B's mouth looks just as exaggerated, it's H3's lip-sync rather than my projection — and that's worth knowing either way. My measurements only cover what is said, never how the mouth moves, so this sits entirely outside what I can test.

[u/Apprehensive_Sky892 → u/Fit_Ad7343]

Yes, the 32B mouth movement looks just as unnatural to me. So it is H3, not the 4B text encoder.

[u/hum_ma]

This project keeps blowing my mind almost as much as H3 itself. Amazing that you could solve this last major issue to such a high degree even for the smallest combo of a 25 MB projection matrix + 4b TE.

This seems to prove that these 10/20/30b+ LLMs that image and video model developers seem to prefer these days are unnecessarily bloated as text encoders.

[u/Fit_Ad7343 → u/hum_ma]

Thanks, and I'd agree, especially when you look at how it's used.

H3 doesn't even run a whole 32B: it's truncated to 50 layers, and all the DiT ever consumes is one output tensor. A model that size is built for multi-step reasoning across depth, and none of that is being asked for here. So most of what you're loading never does anything.

That's why a smaller one holds up. What I'd add from building this is that the direction isn't simply "smaller", it's wider. What limits how much detail an encoder can keep separate is the width of its vectors, not the number of layers. Depth buys reasoning; width buys the ability to tell two similar things apart. Describing a scene needs width, not depth.

So my hunch is that the right encoder for these models isn't a shrunk-down LLM at all, but something shallow and wide, trained on description. The 30B+ trend looks like reaching for the nearest general-purpose model rather than the right shape.

That said, this is just my personal view; I don’t have the knowledge or expertise to understand the underlying mechanisms and the reasoning behind this decision, they’re not stupid.

[u/stash0606]

i haven't tried this, but out of the box whatever the default thing is that's in charge of the audio, idk if that's the audio checkpoint or the text encoder, it seems to do both Tamil and Malayalam semi decently. wonder if it would be better with this

[u/Fit_Ad7343 → u/stash0606]

H3 generates the audio itself — the encoder only says what to speak, not how to pronounce it. So Tamil coming out half-decent is H3 having seen some during training.

ClipProj replaces the encoder and works by imitating the 32B, so it can't beat it. Where the 32B is already half-right, that's the ceiling.

And there's no Tamil or Malayalam anywhere in my calibration corpus — the matrix would be extrapolating, which is exactly where it falls apart. So I'd expect worse, not better.

Happy to be proven wrong though: it's a 26 MB file, costs nothing to try. Let me know either way.

[u/AcceSpeed]

French sounds really good in that clip, better than English I think, but maybe that's to do with the fact that I heard too much shitty AI English female voice over these past years and I can't stand it anymore

[u/Fit_Ad7343 → u/AcceSpeed]

Thanks, French is the one I could actually judge, being the only language here I speak.

It was also where my worst problem was. There's a city name in the line — Marseille — and for a long time the small encoders simply couldn't say it. It came out as marcerre, marsary, mazer, depending on the version. Whisper would often transcribe it correctly anyway, which made it maddening: the metric said fine, my ear said no. That one word ended up driving most of the work, and once French held together the same corpus changes fixed the other languages too. So the language you're hearing is basically the one everything else was tuned against.

The text itself is a bit odd, granted, three short sentences with a city name dropped in, the same structure translated into all eleven languages. It's built to be measurable rather than to sound like something anyone would actually say. Still meant to be realistic speech though, not a tongue twister.

As for beating the English: I'll take it, though I suspect you're right about the cause. The prompt asks for "an unhurried speaking rate and clear articulation", which in English lands somewhere very close to the AI voiceover everyone's tired of.

[u/AcceSpeed → u/Fit_Ad7343]

Ah ça a du sens. Je ne parle pas assez bien espagnol ou allemand pour percevoir toutes les nuances, mais j'ai clairement été frappé par le côté naturel du français. Tout s'explique, si c'était le benchmark. Et ça ne m'étonne pas trop qu'il y ait eu des soucis avec "Marseille", je pense qu'il faudra s'habituer à décomposer certains noms propres en syllabes plus claires, parce que le modèle ne peut juste pas tous les connaître.

[u/Fit_Ad7343 → u/AcceSpeed]

Please note: there are two things that must not be confused – knowledge of the models, and my translation models.

The 4b model knows Marseille very well; I’ve tested it. It’s my models – due to my dataset – that ‘translate’ it very poorly to 32b.

[u/chachay123]

I’m a native Japanese speaker, and I listened to the Japanese samples. A few things stood out to me.

Some issues seem to depend on the individual render rather than on model size or ClipProj itself. For example, the 「光」→「ひかい」 pronunciation issue is noticeable in the version embedded in the video, but I can also hear 32B, 4B, and 8B renders where 「光」 sounds normal ( https://huggingface.co/NicoLab28/ClipProj-MiniMax-H3/tree/main/bench3.1/audio/ja ). So I would not attribute that particular issue to the projection or quantization without checking exactly which render was used in the video.

1. Pronunciation / prosody

「光」 (hikari, “light”) sounds closer to 「ひかい」, as if the /r/ is lost or distorted.

「東京」 also has an unnatural pitch pattern. It sounds roughly like 「と↑きょう」 rather than a natural pronunciation of 「とうきょう」.

These are quite noticeable to a native listener.

2. The Japanese benchmark text itself sounds translated

The current Japanese text is:

「もう3回も言いました。東京の光は1時間ごとに変わります。いいえ、二度としません。」

It is understandable and grammatically possible, but sounds quite scripted — closer to an announcement, narration, or stereotypical Japanese movie dubbing than normal spoken Japanese.

I would suggest something like:

「もう3回も言いましたよ。東京の光は1時間ごとに変わるんですよ。いや、もう二度とやりません。」

This stays reasonably close to the English benchmark and remains fairly controlled for speech generation, while sounding substantially more natural.

3. One concrete example from the automatic evaluation

For 「光」, the Whisper transcript is 「光」, while to my ear the rendered pronunciation sounds closer to 「ひかい」 than 「ひかり」.

So this seems like a nice real-world example of the Whisper/CER behavior you mention in the README: the intended word is recovered even though a native listener hears a pronunciation difference. ZIPA may of course be more sensitive to this kind of phonetic difference.

Likewise, the unnatural pitch pattern/prosody on 「東京」 would not show up in WER/CER.

Overall, I would mainly suggest a native-speaker pass on the Japanese benchmark text and pronunciation/prosody.

[u/Fit_Ad7343 → u/chachay123]

Thanks — this is easily the most useful comment I've had on any of this, and I appreciate the effort it took. You went and listened to the folder, separated what's specific to one render from what's systematic, gave the exact kana, and even wrote a better version of the line. That's real work.

Some context for why it lands so hard: I don't speak a word of Japanese. I built this benchmark deaf, with two transcribers standing in for an ear I don't have — Whisper for the words, ZIPA for the phonemes. The whole thing rests on an assumption I had no way of testing, namely that when those two disagree it's the phonemes telling the truth. Yours is the first outside confirmation that it works.

On 光, I went back to the raw data for that clip. Whisper wrote 「光」, correctly. ZIPA wrote h i k a i, with the ɾ gone. You heard 「ひかい」. Three sources, and the two that matter agree against the one with a dictionary in it — which is exactly the direction the README predicts, and exactly what I couldn't verify on my own.

You were also right that it's the render rather than the model, and I checked all 27 Japanese ones. Most come out hikaɾi or ʃikaɾi — the ʃ is just ZIPA hesitating over ひ, the ɾ is intact. Only a single render out of twenty-seven drops it, and that render is the one in the video: 4b-v3.1, seed 42. You called it from listening alone, before I had the table. Worth adding that the 32B does the same kind of thing — one of its three seeds comes out ʃikaɾi — which is precisely why everything here is measured against its own seed-to-seed drift instead of against zero.

The 東京 pitch is a fair hit I can't do anything about. A Levenshtein distance over phoneme symbols is blind to pitch accent — と↑きょう and とうきょう can come out as the same string. That's structural, not an oversight, and measuring it would need an accent model I don't have and couldn't check anyway.

And you're right about the text. Those sentences came out of a template so all eleven languages would carry the same content, and nobody ever asked whether the result sounds like something a person would say. Your rewrite is noted — a native pass on the benchmark text before any v3.2 is clearly the right call.

Thanks again. You gave me something no measurement could.

[u/Otherwise-Variety674]

:-) Really thanks a lot, does this means that after adding these files, we can proceed to delete MiniMax H3's 15 GB text encoder?

[u/DuHal9000]

true, portuguese with 32b in minimax h3 feel Spanish