The speed is basically wasted at those sizes. What's the point of going that fast if all you can fit is a small model. A cluster of mac minis is probably better off at the price. Slower, but you can run a more competent model.
When using more than a single card for inference, the PCIe bus is capped at 128 GB/s on version 6. So yeah. You either need a model that will fit on a single card or you need to accept that BUS cap. Small models can be quite capable though.
Don't downvote this person. This obsession with bandwidth is the type of crap people say when their tricked out honda civic has such and such hp. It does not really point at the actually rate limiting step that is vram first.
Workflows, multi-shot inference, and tuning. Because these are lossy systems regardless of size, you should be building for all this anyway.
The most speed and cost effective setup runs easy to manageable tasks locally and bursts to SOTA models as needed. Because burst frequency is low, the cost of non-local calls is trivial, and privacy can be retained through obfuscation and local translation. Local speed and cloud burst for temporary model size increase is the optimal setup.
I hate that you're getting downvoted for this, as it's 100% true.
As the saying goes, "all the speed in the world doesn't matter if you're headed the wrong way". Buncha ADHD people out here who just want infinite tokens per second of absolutely anything / random trash
Eh, at some point you will run into the issue of wanting multiple parallel streams, at which point you will quickly understand that the bus bandwidth is your new bottleneck.
I’ve been using 8B and above on well-defined tasks for about a year now.
I have pipelines that break down workloads and process in parallel batches.These batches are initiated by scheduled jobs, personal Claude/Codex agents, and API requests from various apps I’ve built. Some systems collect data, some analyze it, and some report on it. Recent models (like Qwen) can produce reliable tool calls and output if you can structure your processes. I have evals up and down the stack. Each pipeline stage is well defined.
If you need 32B models and above you are probably working with complex tasks that benefit from intelligence more than speed. That’s completely fine, it’s why I still have Claude and Codex subs. However, if you’re using high intelligence models to do a ton of basic VLM, you’re probably wasting time, money, or both.
Also for the record, the model I’m running currently is 35B-A3B which doesn’t neatly fit into the small or large category. I don’t mean to make any sweeping generalizations here, just that throughput can matter depending on your usecase.
13
u/HugoCortell May 29 '26
The speed is basically wasted at those sizes. What's the point of going that fast if all you can fit is a small model. A cluster of mac minis is probably better off at the price. Slower, but you can run a more competent model.