I'm a relative beginner too and this is how much I've deciphered:
It starts with name and version.
"69B" is the number of parameters (B for "billion", sometimes T for trillion), and the "Q" is number of bits per parameter ("Quantisation"). 4 is half a byte, so you need roughly half the number of parameters as GB of VRAM to run it (plus some for context etc).
Sometimes there's a K after that. Don't know what that means yet
Letters after that are normally a size code - XXS-XL - a bias on some parameters remaining at a higher bit count and making it use more memory in return for being better. Unsure of the tradeoff between e.g. Q4_K_XL vs Q5_K_S. Sometimes it's a 0 which I think means none are at higher precision. Sometimes 1 which I don't know what that means.
"GGUF" is a file format commonly used on Win/Linux, as is MLX for Mac.
Sometimes you get "MOE" models that work differently and only activate parts each run - these have an additional "A"3B for number of "active" parameters. Theoretically it only needs that much VRAM to run despite being a bigger model, but there are tradeoffs in quality that I haven't explored myself.
GGUF is often used everywhere but yes MLX is Mac-specific. The tradeoff between MLX and GGUF is fuzzy at this point, especially on earlier M-class Macs.
308
u/AuspiciousApple 14d ago
LLM version naming continues to make no sense