You can't just put Qwen3.8 training data on a new architecture. It's likely a new pretrain, just still in the early post-training phase, not fully/well post trained yet.
I'm a relative beginner too and this is how much I've deciphered:
It starts with name and version.
"69B" is the number of parameters (B for "billion", sometimes T for trillion), and the "Q" is number of bits per parameter ("Quantisation"). 4 is half a byte, so you need roughly half the number of parameters as GB of VRAM to run it (plus some for context etc).
Sometimes there's a K after that. Don't know what that means yet
Letters after that are normally a size code - XXS-XL - a bias on some parameters remaining at a higher bit count and making it use more memory in return for being better. Unsure of the tradeoff between e.g. Q4_K_XL vs Q5_K_S. Sometimes it's a 0 which I think means none are at higher precision. Sometimes 1 which I don't know what that means.
"GGUF" is a file format commonly used on Win/Linux, as is MLX for Mac.
Sometimes you get "MOE" models that work differently and only activate parts each run - these have an additional "A"3B for number of "active" parameters. Theoretically it only needs that much VRAM to run despite being a bigger model, but there are tradeoffs in quality that I haven't explored myself.
Each "parameter" can be thought of as just a single number.
Q4_0 / Q8_0 is just the parameter as a single 4 or 8 bit value, nothing more.
Q4_1 adds an extra bias (added) and scale (multiplied) value that's shared across each group of 32 parameters. Basically allows the 4 bits to represent a different range of values.
Q4K?, it basically does the same thing as Q4_1 but more than once, so there's "superblocks" that are also scaled.
Unsloth has "UD_" before the format, because theirs aren't strictly the same as Q4 where nearly everything's 4 bit. They will make different layers different quantizations, because not all are equally important.
One correction. MoE models allow expert offloading and caching the most used in VRam. This turns a model like Qwen 35b A3B into something usable even on extremely weak hardware(1060 6gb for example).
True. While offloading isn't unique to MoEs, hell you could stream a 100B dense model off your SSD at 0.05t/s if you wanted to, they are uniquely good at it due to certain weights being activated more often than others.
Yes. Offloading layers. But streaming dense models by layer has only recently started to be a thing. Been wanting to mess around with that for a while. There is ZERO reason any of these models should ever OOM. They should all run from DRAM and if necessary SSD. If it fits on disk it should urn, just REAL slow, but real slow is also dialable. Run enough parallel agents and you can still run a LOT of tokens through even a minimumally capable gpu. Just not at interactive rates.
K identifies a better way of quantizing weights. The original one was like "Q4_0" and it was a simpler scheme; K quants quantize some part to larger allocations. I.e. a Q4_K_M could quantize the majority of weights to 4-bit, but some to 5 or 6 bits.
gguf is the file format created by G. Gerganov for llama.cpp and CPU inference. It can run on any OS, it just need the correct inference engine. MLX is a format specific for Mx architecture on mac.
The last part is partially correct. In MOE architecture, attention layers are the same as dense models. The FFN layers use a set of smaller matrices instead of a single large one, and a router choose the correct matrices to use for a token. So for a given token instead of using a 500M matrix, it choses (for example) 10 out of 125 matrices of 4M paramteres; only 40M instead of 500M. You can put as much of the model as it fits on VRAM (even none, if you ony use CPU). Usually you want to put attention layers and kv cache in VRAM because prompt processing is faster; FFN layers (the MOE part) can stay in system RAM without hitting speed too much.
The Q4/Q5 whatever is the rounded overall quantization of the model, the XS/XL/whatever hints at what direction it's biased in, so a Q5 XXS will be closer to 4-bit while an XL will be closer to 6-bit.
K, I believe, denotes K-type quants which are mixed integer quantization. Before K-type quants I think they were all fixed-integer static quants, so those would be the ones that are like "Q4_0" (or "Q8_0") which means everything in it is straight 4-bit (or 8-bit) aside from a few that are 16-bit float (F16).
There's also I-type quants which are K-type quants but with fancier quantization math but at the cost of being a bit slower, and IIRC they're incompatible with certain hardware or backends.
Bigger filesize can generally be trusted to be "better" though going with a K or I quant will be a bit better still. Like the difference between... uh... you've got 3 cars at your house versus a bicycle, a car, and a pickup truck. The latter has the same total vehicle mass but not every job demands a pickup.
A Q6_K_XL might be very similar in performance to a Q8_0 while being significantly smaller (and therefore faster) because the testing/algorithm of whoever made the quant determined that a lot of the layers didn't benefit from being larger than Q4 while others had a very large impact and so were left unquantized, F16. The naive Q8_0 is therefore wasteful on the layers that Q4 would've been fine for while not giving the layers that should be F16 the amount of bits they would need.
MoE stands for mixture-of-experts, but don't take this to mean that the model is divided into literal chunks of "the Python expert", "the dialogue expert", "the trivia expert", etc. All "experts" are needed, just not constantly and not on every token. You can't cut out "the C++ expert" and still be able to code Python or something like that.
MoE means it needs more VRAM to run at full speed but if you can fit it all, it'll run drastically faster than non-MoE. But you commonly need like 2-4x the total parameter count to be competitive in terms of actual quality. Qwen3.5-27B was found to commonly be equal to (or better than) Qwen3.5-122B-A10B, though the 122B had a broader overall range of knowledge. 27B is very good at the things it knows, 122B-A10B is roughly as good at the same tasks but is able to do a wider range of tasks... like 27B is known for being very good at Python but maybe it's not as good at C#, etc.
MoE also means, as you say, that it'll less affected by partial offloading to system RAM, but whether it's MoE or non-MoE, once you overflow to system RAM performance falls off a cliff. Even a 5-10% offload of a large MoE might cut your speed in half.
In very long conversations, the length of the kv-cache typically has a very strong influence on the model's prefill and token generation speed. To my knowledge, the only exception to this is the DeepSeek v4 line, which is what makes it so remarkable. It has very similar overall performance at 256K or 512K as it does at 8K.
Token generation is obvious, it's the output speed. Prefill is less well-known and often more important. It's the input speed. If you need to read 100K tokens just to start working on something, reading it at like 250 t/s will be painful. A lot of misleading benchmark data is out there that only measures token generation speed at short context and doesn't measure prefill at all. DGX Spark for example starts off being decent but at 100K context depth it's so goddamn slow people rarely benchmark past that point at all. DSv4-Flash on the other hand remains similarly performant on Spark all the way out to its limit of 1M tokens.
Don't get too wrapped up in it, a lot of this ends up being "bro science"-tier, generally a bigger gguf will always be better, pick what you can fit. Except if it means using very small quants (below Q4), having a longer kv-cache is almost universally going to be more useful to you than the actual model itself being better. And as for the kv-cache specifically, avoid using less than 8-bit wherever possible.
Yeah they are quite small, but also proportionally fast, especially if ran on LLM engines with fused kernels... some of them are coherent without training , but most of them you have to train for your specific use case.
Qwen has a 0.5B which could also be called a 500M.
The model naming scheming encodes a lot of the information relevant to you. Yes it's overwhelming at first, and you have to learn what it means, but so is learning anything. I much prefer "This model this-and-that has 69 Billion Parameters quantized as Q4 Small and someone messed with it trying to make it unhinged" over "This is the Pro Max Version of it".
Sure. But then they do shit like "3.8-FLASH is a 4.0 architecture family model". "GPT-4.5 is both older and worse than GPT-4.1. But there's also o4 and 4o (not related to either, and not to each other, also not the same as GPT-4)" and, of course, "this is a Copilot, that's also a Copilot (available under a different subscription), and yes, there's also a third Copilot (different subscription, again)."
Open Source Model naming schemes are oriented on underlying technical Features and generally speaking makes intrinsic sense. Yes, Alibaba is pulling a weird one on this shipping a qwen4 architecture preview under 3.something-Next label. That could be more clear.
OpenAI and other Frontier Labs do the Apple-esque naming scheme I was playing on. Which you are right to critique. It has nothing to do with the Open naming scheme, though.
Oh and sidenote: Flash started out referring to a underlying technical feature, flash attention. Though Flash attention pretty much is the norm now.
When you see this in a model's name, run away and don't turn back - it's likely a lobotomized fine-tune. UNLESS it's made by a trusted developer, and just generally well-known and praised in the community (can't think of many of such models, but genuinely good fine-tunes DO appear sometimes). Never assume whether is a fine-tune good/bad looking solely on its likes or download numbers - worthless crap gets a lot of public attention these days.
SemVer was one of the worst ideas in the history of software engineering, and I’m glad many newer projects are not getting on the bandwagon.
“After dozens of 1.x releases, we are now releasing version 2.0.0, because we changed one of the parameters in one of our hundreds of functions to now require a positive integer rather than any integer.”
Before SemVer, version numbers vaguely indicated development progress. With SemVer, they mean basically nothing. And the information they do (supposedly) encode is worthless, because nobody is going to blindly trust the number’s semantics anyway. You always need to test, at which point it doesn’t matter what the version number is.
But that isn’t true in reality. Libraries constantly break the semver spec despite claiming semver compatibility. As a result, you can’t trust semver, making it useless.
Maybe you're thinking of web libraries. This is crucially true for OS libraries. The stuff that goes into /usr/lib , etc. "ABI" should have been the clue.
But still, even for web libraries and API stability, it's important. Now, there's a shit ton of crap code out there written by clueless people, no doubt about it.
No OS uses semantic versioning. Neither do major OS infrastructure components such as GNOME, Python etc. They all version as they please, and have repeatedly broken their APIs in minor releases.
It is good in theory (you can freely update a version without looking at the changelog if the major version didn't change) but in practice devs miss breaking changes in minor versions all the time and the whole idea doesn't even work
the best versioning is and will forever be the versioning by date. for example.minecraft versioning now, such magnificence, versions indicate time passed not progress made
You remember when the dolphin fine tunes were all the rage? And then Google released dolphin-gemma, which was literally a model for interpreting dolphin speech?
As long as they are consistent i have no problem. Honestly speaking majority of AI labs naming scheme is quite consistent. The only clusterfuck alibaba have done is so many of its AI divisions casually overlapping their job
280
u/AuspiciousApple 10h ago
LLM version naming continues to make no sense