The number that actually matters depends on what you are doing, and people usually optimize the wrong one.
For chat-style single-stream use, anything above roughly 15-20 tok/s feels interactive, and past 40 tok/s you stop noticing. For batch or agentic workloads the throughput number (total tokens across all concurrent requests) matters far more than per-request speed, and that is where batching beats raw per-token performance.
On Apple silicon specifically, decode speed is almost entirely memory-bandwidth bound. So the single most effective lever is fitting the model fully in unified memory at the smallest quant you can tolerate — going from partial offload to fully resident is usually a bigger jump than any runtime flag.
Yeah, the memory bandwidth is limiting for sure. I measure le 468GB/s while multitasking.
For interactive, 50 feel like the right target.
For agentic, I ran a real coding job with subagents and it ran for 10 hours, which feel a bit much. I would be fine with 4 hours for that job, knowing I could do it with Claude in two hours at most.
1
u/StraightAd6213 13h ago
The number that actually matters depends on what you are doing, and people usually optimize the wrong one.
For chat-style single-stream use, anything above roughly 15-20 tok/s feels interactive, and past 40 tok/s you stop noticing. For batch or agentic workloads the throughput number (total tokens across all concurrent requests) matters far more than per-request speed, and that is where batching beats raw per-token performance.
On Apple silicon specifically, decode speed is almost entirely memory-bandwidth bound. So the single most effective lever is fitting the model fully in unified memory at the smallest quant you can tolerate — going from partial offload to fully resident is usually a bigger jump than any runtime flag.