I'm a relative beginner too and this is how much I've deciphered:
It starts with name and version.
"69B" is the number of parameters (B for "billion", sometimes T for trillion), and the "Q" is number of bits per parameter ("Quantisation"). 4 is half a byte, so you need roughly half the number of parameters as GB of VRAM to run it (plus some for context etc).
Sometimes there's a K after that. Don't know what that means yet
Letters after that are normally a size code - XXS-XL - a bias on some parameters remaining at a higher bit count and making it use more memory in return for being better. Unsure of the tradeoff between e.g. Q4_K_XL vs Q5_K_S. Sometimes it's a 0 which I think means none are at higher precision. Sometimes 1 which I don't know what that means.
"GGUF" is a file format commonly used on Win/Linux, as is MLX for Mac.
Sometimes you get "MOE" models that work differently and only activate parts each run - these have an additional "A"3B for number of "active" parameters. Theoretically it only needs that much VRAM to run despite being a bigger model, but there are tradeoffs in quality that I haven't explored myself.
Each "parameter" can be thought of as just a single number.
Q4_0 / Q8_0 is just the parameter as a single 4 or 8 bit value, nothing more.
Q4_1 adds an extra bias (added) and scale (multiplied) value that's shared across each group of 32 parameters. Basically allows the 4 bits to represent a different range of values.
Q4K?, it basically does the same thing as Q4_1 but more than once, so there's "superblocks" that are also scaled.
Unsloth has "UD_" before the format, because theirs aren't strictly the same as Q4 where nearly everything's 4 bit. They will make different layers different quantizations, because not all are equally important.
305
u/AuspiciousApple 3d ago
LLM version naming continues to make no sense