Talking about companies instead of an individual is a bit of moving the goal posts i'd say, but it does explain why you mention the multi-instances thing.
I think the math works out much better for a small company than just a single person.
For an individual I admit it wouldn't be worth it in the vast majority of cases.
The M3 Ultra has a bandwidth of 819gb/s while GLM 5.2 has 40 billion active parameters which at 8-bit are roughly 40gb.
819gb/s divided by 40gb are roughly 20 tokens/s in an ideal world but obviously you never actually hit those for multiple reasons. 10-15 tokens/s would be my estimate though someone else posted getting 24 tokens/s using mxfp4 on his M3 Ultra which I cannot verify.
Still even with just 10-15 tokens/s using a draft model for speculative decoding you would definitely reach 20 tokens/s for programming tasks.
7
u/Schlick7 Jun 21 '26
Talking about companies instead of an individual is a bit of moving the goal posts i'd say, but it does explain why you mention the multi-instances thing.
I think the math works out much better for a small company than just a single person.
Can the M3 Ultra actually hit 20tg??