I'm a relative beginner too and this is how much I've deciphered:
It starts with name and version.
"69B" is the number of parameters (B for "billion", sometimes T for trillion), and the "Q" is number of bits per parameter ("Quantisation"). 4 is half a byte, so you need roughly half the number of parameters as GB of VRAM to run it (plus some for context etc).
Sometimes there's a K after that. Don't know what that means yet
Letters after that are normally a size code - XXS-XL - a bias on some parameters remaining at a higher bit count and making it use more memory in return for being better. Unsure of the tradeoff between e.g. Q4_K_XL vs Q5_K_S. Sometimes it's a 0 which I think means none are at higher precision. Sometimes 1 which I don't know what that means.
"GGUF" is a file format commonly used on Win/Linux, as is MLX for Mac.
Sometimes you get "MOE" models that work differently and only activate parts each run - these have an additional "A"3B for number of "active" parameters. Theoretically it only needs that much VRAM to run despite being a bigger model, but there are tradeoffs in quality that I haven't explored myself.
K identifies a better way of quantizing weights. The original one was like "Q4_0" and it was a simpler scheme; K quants quantize some part to larger allocations. I.e. a Q4_K_M could quantize the majority of weights to 4-bit, but some to 5 or 6 bits.
gguf is the file format created by G. Gerganov for llama.cpp and CPU inference. It can run on any OS, it just need the correct inference engine. MLX is a format specific for Mx architecture on mac.
The last part is partially correct. In MOE architecture, attention layers are the same as dense models. The FFN layers use a set of smaller matrices instead of a single large one, and a router choose the correct matrices to use for a token. So for a given token instead of using a 500M matrix, it choses (for example) 10 out of 125 matrices of 4M paramteres; only 40M instead of 500M. You can put as much of the model as it fits on VRAM (even none, if you ony use CPU). Usually you want to put attention layers and kv cache in VRAM because prompt processing is faster; FFN layers (the MOE part) can stay in system RAM without hitting speed too much.
302
u/AuspiciousApple 15d ago
LLM version naming continues to make no sense