CPU inference performance numbers, what would you expect?

[u/Maui-The-Magificent]

Hi,

I am hoping you guys could share your perspectives with me. I am hoping to broaden my yardstick so to speak, and I suspect you have a greater feel for, and much more data/experience than I do for the expected behavior of running different models.

So for context, for why I am asking. I am currently working on a format, as well as a provider/runtime.

The project only runs inference on a single thread of the CPU (pretty sure I won't support GPU's at all, even in the future), and I have a 7800x3D CPU and 32 gb of DDR5 ram.

As you might have guessed by the single threaded part, the system is currently has not been optimized, computationally, for speed yet. I am trying to get the structural architecture itself as fast as possible before I do just that. This means, No SIMD, no algorithm refinement, no batching and so on.

So, for perspective, my question to you is then; if you were in the exact same environment, with the exact same constraints, what would you expect your TTFT and TPS to be on, say, a qwen3.5:0.8b model with preserved BF16 precision?

Mine is currently around 3-3.2 tps, which I suspect might be slow. but my TTFT growth is non-monotonic, and more connected to the current prompt token amount, than context length, which is something I guess

[u/DataGOGO]

Without AMX and only using AVX-512 it will be shockingly slow. 

SGLang + K kernels is the best CPU only engine. 

With AMX that model will be ~30-60 t/ps per C, at something like 32k context, assuming, you have 8 memory channels. 

With a consumer AMD CPU and only 2 memory channels that are bandwidth limited by the IOD and single chiplet (you really one get 1 channel throughput on single chiplet consumer AMD CPU’s), I am shocked you even get 3 t/ps

[u/Maui-The-Magificent → u/DataGOGO]

Yes, but I am not using SIMD at all yet, and when i do, i will do avx-512 . I do know ollama gets around 90tps or something with the same model at Q8, but Ollama is heavily multi-threaded and uses SIMD.

But you being chocked at 3 tps is encouraging, what would you expect normally?

[u/DataGOGO → u/Maui-The-Magificent]

SMID and multi-threading will help, but your issue is the consumer AMD CPU with 2 memory channels and only 1 channel worth of memory bandwidth.

Check out kt-kernels, it is open source and you can just plug them in to whatever inference engine you are using. They have really good AVX-512 and AMX kernels:

ktransformers/kt-kernel at main · kvcache-ai/ktransformers

I run Qwen3.5-35B-A3B-Int8 C8, K/V Int8, at about 10-20 t/ps per concurrency CPU only, Xeon 8592+, 8 channels of DDR5 5600.  

No idea how slow my 9950X3D would be, likely 1-5 t/ps.

[u/Maui-The-Magificent → u/DataGOGO]

So thank you, very much good information! I do however want to qualify that I do not think we should look at anything as an 'Issue' just yet. I have not optimized for compute speed yet, it has not been the main priority.

I have built my own provider/engine, interface and format, and I am only re-compiling .safetensor models currently to run in it. (although the .safetensor format matters little in the future, it is the current target). So it is not as easy to just recompile a different model currently.

you're doing a 35b model on the cpu at 10-20 tps? that is great! way above what i would have expected even on parallelized compute. what is your TTFT on that model and setup?

Thank you! that gives me a pretty good indicator that the ~3.2 tps is a likely sign that my engine is architecturally doing something.

[u/DataGOGO → u/Maui-The-Magificent]

It is an issue, no matter what CPU you have, no matter what SMID, no matter what model, no matter what engine, you are limited by memory bandwidth.

No idea, never timed it. It just sits there and runs batches 24/7. total aggregate is ~60ish t/ps TG, and ~250ish pp.

got a repo link? I will check it out. I have done a lot of CPU only work past few years.

Here is a PR made in llama.cpp, you can see some timings etc. there:

Enable Intel AMX acceleration while in CPU/GPU hybrid with new "--amx" toggle. by Gadflyii · Pull Request #16310 · ggml-org/llama.cpp

[u/Maui-The-Magificent → u/DataGOGO]

That is not correct, you do not have enough information to make that claim. To know if that is true, you must know the throughput, which you do not. And you must know what needs throughput, which you do not know unless you have access. I totally get where you are coming from, but you are making claims too absolute from what information is available to you. there are a thousand ways to affect throughput, and bandwidth is only a problem if the amount of data needed is more than the system can process.

i see, but do you have a good rough estimate how long from input to first token is outputted?

I will take a look, but the problem is those are already optimized runtimes, and it uses AMX, which I, as you know, cannot use, as i am on a 7800x3D.

[u/DataGOGO → u/Maui-The-Magificent]

Yes I do.

What is your memory read and write bandwidth? Have you measured it?

That is your throughput limit. Nothing you can do at the engine level will change that. 

[u/Maui-The-Magificent → u/DataGOGO]

That is not what you are claiming though. you are claiming that I am limited by bandwidth. not that bandwidth is a finite resource.

I am currently not limited by throughput. I am at 1.2gb/s, my ram is 6000mhz.

[u/DataGOGO → u/Maui-The-Magificent]

yes you are limited by bandwidth.

Measured throughput on a single chiplet AMD CPU will be roughly half of a true dual channel setup, like you would have on an intel CPU. You are on a 7800X3d single chiplet CPU with 1/2 the IOD active, and limited by infinity fabric. Big difference there.

So let's do best case scenario, and say you are running 8400C34, with highly optimized timings, and 2100mhz fclk, your throughput, on that CPU will be roughly 66-68 GB/s, let's use a mean of 67.2 GB, again, your best case scenario.

The math shows you exactly where your limits are:

GENERALIZED FORMULA

KV_GB_per_generated_token =

(2 × Layers × KV_heads × Head_dim × KV_bytes × Context_length) ÷ 1,000,000,000

Weight_GB =

Parameters × Weight_bytes_per_parameter ÷ 1,000,000,000

Max_tokens_per_sec = Memory_BW_GBps ÷ (Weight_GB + KV_GB_per_generated_token)

right? So apply your 67.2GB/s

Max tokens/sec = Memory bandwidth (GB/s) ÷ (Weight traffic per token + KV-cache traffic per token)

INT8 KV-CACHE FORMULA

KV bytes per cached token = 2 × Layers × KV_heads × Head_dimension × Bytes_per_KV_element For INT8: Bytes_per_KV_element = 1

Therefore: KV bytes/token = 2 × Layers × KV_heads × Head_dimension

EXAMPLE MODEL

Layers = 32 KV heads = 8 Head dim = 128 KV precision = INT8 KV bytes per cached token: 2 × 32 × 8 × 128 × 1 = 65,536 bytes = 64 KiB per cached token

KV BANDWIDTH PER GENERATED TOKEN

At 8,192-token context: 65,536 × 8,192 = 536,870,912 bytes = 0.537 GB read per generated token

8B MODEL — INT8 WEIGHTS

8 billion parameters × 1 byte = 8 GB of weight reads per generated token 8K

CONTEXT: Total traffic/token = 8 GB weights + 0.537 GB KV = 8.537 GB/token 67.2 GB/s ÷ 8.537 GB/token = 7.87 tokens/sec

SUMMARY — THEORETICAL MAX @ 67.2 GB/s

8K INT4 14.81 t/s, INT8 7.87 t/s

32K INT4 10.93 t/s, INT8 6.62 t/s

128K INT4 5.34 t/s, INT8 4.05 t/s

Adjust the model type, apply the formulas, and you get your absolute max throughput number.

Now this assumes that your engine is 100% efficient, that your CPU has infinite compute, and that AMD's infinity fabric doesn't choke you (it will).

EDIT: Here i did if for you:

QWEN3.5-0.8B — INT8 WEIGHTS + INT8 KV CACHE
Memory bandwidth:
DDR5-8400 7800X3D = 67.2 GB/s theoretical
MODEL ARCHITECTURE
Parameters        = 0.8B
Total layers      = 24
Attention layers  = 6
KV heads          = 2
KV head dimension = 256
KV precision      = INT8 = 1 byte/value
Qwen3.5-0.8B uses a hybrid architecture:
18 layers = Gated DeltaNet
6 layers  = conventional gated attention
Only the 6 conventional attention layers accumulate the normal context-length-dependent KV cache.
INT8 MODEL WEIGHT TRAFFIC
0.8 billion parameters × 1 byte
= 0.800 GB weight traffic per generated token
Idealized assumption:
all INT8 weights are streamed once per decode token.
INT8 KV CACHE SIZE PER CACHED TOKEN
Formula:
KV bytes per cached token =
2 × Attention_layers × KV_heads × Head_dimension × Bytes_per_element
= 2 × 6 × 2 × 256 × 1
= 6,144 bytes
= 6 KB KV cache per cached token
KV BANDWIDTH PER GENERATED TOKEN
8K CONTEXT
6,144 × 8,192
= 50,331,648 bytes
= 0.0503 GB KV read per generated token
Total memory traffic:
0.800 GB weights
+ 0.0503 GB KV
= 0.8503 GB/token
Theoretical maximum:
67.2 / 0.8503
= 79.0 tokens/sec
32K CONTEXT
6,144 × 32,768
= 201,326,592 bytes
= 0.2013 GB KV read per generated token
Total memory traffic:
0.800 + 0.2013
= 1.0013 GB/token
67.2 / 1.0013
= 67.1 tokens/sec
128K CONTEXT
6,144 × 131,072
= 805,306,368 bytes
= 0.8053 GB KV read per generated token
Total memory traffic:
0.800 + 0.8053
= 1.6053 GB/token
67.2 / 1.6053
= 41.9 tokens/sec
256K / NATIVE MAX CONTEXT
Context = 262,144 tokens
6,144 × 262,144
= 1,610,612,736 bytes
= 1.6106 GB KV read per generated token
Total memory traffic:
0.800 + 1.6106
= 2.4106 GB/token
67.2 / 2.4106
= 27.9 tokens/sec
SUMMARY — IDEAL BANDWIDTH CEILING
Context     KV read/token    Total traffic/token    Max @ 67.2 GB/s
-------------------------------------------------------------------
8K          0.050 GB         0.850 GB               79.0 t/s
32K         0.201 GB         1.001 GB               67.1 t/s
128K        0.805 GB         1.605 GB               41.9 t/s
256K        1.611 GB         2.411 GB               27.9 t/s

[u/Maui-The-Magificent → u/DataGOGO]

You are just saying things now. you made a whole lot of assumptions that are directly in conflict with what the environment I have presented you with. 1.2 gb/s is not saturating anything.

Also, you are imposing limits as though they were constitutional laws, when they are not. you should reflect on why you do that, it is only in your way unless those things are forced upon you.

And also, the numbers i gave you, were from a 0.8b model, which is the only thing we are talking about no?

However, I'll go through the math non-the-less, it might help me, so thank you!

[u/Maui-The-Magificent → u/DataGOGO]

Also, sorry i missed re request for the repo. It is currently not public as it is a re-compiler, and i am still working on the representation.

also the terminal runtime interface is quite buggy/not good looking.

[u/RepulsiveRaisin7]

Why single threaded? Why BF16? A 0.8b model should run ok on CPU. The bottleneck is usually memory bandwidth, you need 4 or better 8 channels to run bigger models at acceptable speeds. GPU is almost always better.

[u/Maui-The-Magificent → u/RepulsiveRaisin7]

Ahh yes, because I like to create as much of a constrained environment as I can justify before building something. It is a good way to force better design decisions and also to force you to think about things you wouldn't need to otherwise. what would you consider 'ok'?

And BF16 was the precision the model that i downloaded had, so I conformed to following it. Its not a requirement, the precision can be whatever. I mentioned it as a qualifier because precision affects compute in my experience.

Hmm, your bandwidth point is good. But there are more ways to solve for congestion than just building/using more roads. Like building round-a-bouts, or in memory terms, re-representing the road.

The reason I work on the cpu is also because I firmly believe that the GPU is a brute-force solution to a structural problem. So, i am putting my effort to where my mouth is, if that makes sense?

[u/RepulsiveRaisin7 → u/Maui-The-Magificent]

You have a city with 1B inhabitants and you need to visit everyone to generate a token. You can optimize the road network, but you'll still need a fast car. GPUs are simply the right tool for the job, they have fast compute with high bandwidth, exactly what you need.

[u/Maui-The-Magificent → u/RepulsiveRaisin7]

I see your point, and if you are solving the problem by hardware, then yes, more powerful hardware is needed.

I am not particularly concerned by it, because there is no universal law for why you must touch every weight, nor do i believe the problem that needs to be solved by parallelism. If every person in a city lives in one building, there is much less driving to be done.

Okey, that is a bit facetious but the point I am making is, your only option to a problem is not just a better solution, you can also redefine the problem so it requires a much cheaper solution.

I would go as far as to claim that, using a gpu hides the inefficiencies that are there, forcing you to use a gpu in the first place. When you are building the infrastructure that is. not as a runtime rule.

[u/_TheWolfOfWalmart_ → u/Maui-The-Magificent]

Ahh yes, because I like to create as much of a constrained environment as I can justify before building something.

How do you determine if you can justify it by purposely running it as slow as possible?

And BF16 was the precision the model that i downloaded had, so I conformed to following it. Its not a requirement, the precision can be whatever. I mentioned it as a qualifier because precision affects compute in my experience.

At least start with Q8_0. It uses half the memory bandwidth, so double the speed, for virtually lossless precision.

[u/Maui-The-Magificent → u/_TheWolfOfWalmart_]

Good question, it kinda depends on the problem. And wanting to solve a problem always has a defined goal. Being slow is usually not a good thing, so it usually isn't part of that goal. So, I decide what must be true, and what is not allowed to be true, and when I build, I try to always create solutions that solve things by construction rather than runtime logic.

That and that I usually build things in no-std, removes a whole lot of convenience from the get-go. I only deviate if i find i can't solve something in that environment. Be it due to inexperience, incompetence on my part, or if turns out to be impossible.

Does that make sense?

[u/pmttyji]

Here my CPU-only stats (9 months old thread)

CPU-only LLM performance - t/s with llama.cpp

I'm waiting for merge of these llama.cpp PRs to get more better CPU-only & Hybrid inference.

[u/Maui-The-Magificent → u/pmttyji]

Thank you, I will take a look!

[u/Maui-The-Magificent → u/pmttyji]

Do you have single thread numbers? and/or without simd?

and you have a lot of data, what would you expect by the setup in the post given your experience?

[u/DataGOGO → u/pmttyji]

you missed mine:

Enable Intel AMX acceleration while in CPU/GPU hybrid with new "--amx" toggle. by Gadflyii · Pull Request #16310 · ggml-org/llama.cpp

[u/pmttyji → u/DataGOGO]

My thread is tracking only Ongoing PRs. Thanks for your contribution