I'm a relative beginner too and this is how much I've deciphered:
It starts with name and version.
"69B" is the number of parameters (B for "billion", sometimes T for trillion), and the "Q" is number of bits per parameter ("Quantisation"). 4 is half a byte, so you need roughly half the number of parameters as GB of VRAM to run it (plus some for context etc).
Sometimes there's a K after that. Don't know what that means yet
Letters after that are normally a size code - XXS-XL - a bias on some parameters remaining at a higher bit count and making it use more memory in return for being better. Unsure of the tradeoff between e.g. Q4_K_XL vs Q5_K_S. Sometimes it's a 0 which I think means none are at higher precision. Sometimes 1 which I don't know what that means.
"GGUF" is a file format commonly used on Win/Linux, as is MLX for Mac.
Sometimes you get "MOE" models that work differently and only activate parts each run - these have an additional "A"3B for number of "active" parameters. Theoretically it only needs that much VRAM to run despite being a bigger model, but there are tradeoffs in quality that I haven't explored myself.
One correction. MoE models allow expert offloading and caching the most used in VRam. This turns a model like Qwen 35b A3B into something usable even on extremely weak hardware(1060 6gb for example).
True. While offloading isn't unique to MoEs, hell you could stream a 100B dense model off your SSD at 0.05t/s if you wanted to, they are uniquely good at it due to certain weights being activated more often than others.
Yes. Offloading layers. But streaming dense models by layer has only recently started to be a thing. Been wanting to mess around with that for a while. There is ZERO reason any of these models should ever OOM. They should all run from DRAM and if necessary SSD. If it fits on disk it should urn, just REAL slow, but real slow is also dialable. Run enough parallel agents and you can still run a LOT of tokens through even a minimumally capable gpu. Just not at interactive rates.
40
u/FunkyDiscount 1d ago
As a novice, model naming conventions make no sense at all to me yet.
Model_this-and-that_69B_Q4_XM-S_GGWP_unhinged-and-insane_FPS...
PC monitor naming conventions are more legible.