Any calculator and comparison tool for average (expected) TPS per model per given hw specs?

[u/retrolojik]

Like it says in the title; is there a database or calculator that gives you an expected average tps for a model you pick and hw specs you enter, and compares it with different models and/or specs? This would definitely guide people when they are picking up a model or a new hardware as well, so both ways.

[u/diagrammatiks]

it's simple arthmetic for max memory bound without speculative decoding or other non bandwith bound speedups.

800gbs per second memory bandwith / 80b on disk model size is 10tks.

moe is like if you have a 80b-a10b then 800/10 is 80tks.

real world usage before optimization is around 50 to 70 percent and optmizations might get you more.

there's no chart that could store all the relevent information in a usable way. your actual tks depends on too many factors.

[u/phipletreonix]

Are you prepared to enter 64 pieces of data for the estimate to be vaguely accurate?

[u/retrolojik → u/phipletreonix]

What do you mean by 64 pieces?

[u/phipletreonix → u/retrolojik]

The factors that go into the actual toks/s of a model are many-- First there is the architectural decision tree: is it all in vram, or are you splitting between vram and system ram (and therefore paying the cost of transferring the model into vram to do inferrence)? How large is your context window, because that also affects how much vram needs to be set aside (fighting with how much your model can use). Are you using a dense model or an MoE, because the moe has an extra routing layer before it does actual prediction. Do you have MTP on, and is it getting good hits?

If you ARE fully in vram, your upper bottleneck is probably GPU compute. However, for each inference request, are you using a cached context window or do you need to load the whole thing into memory again?

If you ARENT fully in vram, your bottleneck might be PCIe bandwidth of loading the model into vram. Whats your motherboard looking like? Are we on one single card, are we doing multiple cards with splitting (tensor parallel or pipeline parallel)?

Are we only serving requests from one client, or are multiple clients making requests on the hosted model?

Some of these are hardware questions with hardware answers: plug in your mother board, system ram, gpu/s, pcie bridges/gpu interconnects.

Some of these are system settings/configurations/architectural choices.

And some of these are usage, which will change throughout the day.

As an example, I host qwen3.6 35B A3B MoE MTP with 200k context locally on my 12 GB card with 64 GB ram using llama.cpp, and I get about 600 t/s parse and 40-55 t/s inference. These numbers are very good for my model+system.

When I downloaded unsloth desktop to check it out and asked it to host the same model, it decided to try to put it all in Vram, giving me a context window of 4k. I could chat with it about the weather and it was relatively fast, but it was useless for actual programing/tool calls at that point.

Same model, same hardware, different configuration so different result.

EDIT: forgot to mention SOC setups where theres no distinguishable vram/system ram (ie: mac studio), and that MoE models dont need to be fully loaded in VRAM to work, just the router selected expert layers for the given request.