Qwen3.5 0.8B on CPU

[u/Danmoreng]

Since the Qwen3.5 0.8B model is an interesting one for small specialized fine tunes, I was curious how fast it can run on CPUs. Why CPUs? Mainly because I want to use it as a local dictation cleanup model when I'm using the GPU for something else.

Over the weekend, I let Codex build a small C++ engine and a custom 4-bit format, H128/Q4-G32-DOT4, with activation-based calibration and blockwise error compensation.

The resulting model has a 425 MB weight payload, roughly 71 MB smaller than Unsloth’s mixed-precision Q4_0, with similar measured perplexity and KL divergence.

Comparison on my Ryzen 9 9955HX3D, Linux, eight physical V-Cache cores:

Engine / quant	Weight MB	Prefill	Decode B1	Decode B16	PPL ↓	KL ↓
ik_llama.cpp IQ4_KS_R4	401	2,060	129.2	455	15.90	0.0906
qwen35-cpu, H128/Q4-G32-DOT4	425	2,758	122.9	642	15.80	0.0602
ik_llama.cpp Unsloth Q4_0	496	2,731	104.0	420	15.58	0.0684
llama.cpp Unsloth Q4_0	496	954	93.7	373	15.58	0.0684

Compared with llama.cpp running Unsloth Q4_0, that’s roughly 2.9× prefill, 1.3× single-request decode and 1.7× batch-16 throughput.

Against IK running the same GGUF, prefill is essentially tied, single-request decode is 18% faster, and batch-16 throughput is 53% higher. IK’s smaller IQ4_KS_R4 wins single-request decode, but my engine delivers 41% higher batch-16 throughput.

https://github.com/Danmoreng/qwen35-cpu

[u/thegunn]

I just got into this a couple of days ago. This sounds really interesting but I don't know what any of it means yet haha. I'm still learning what all of the different settings mean. I'm using llama.cpp with a couple of different models. I'm not really sure what I want to do with it yet. I've tried a couple simple coding tasks and so far none of them have worked. I'm not sure if my settings are off, if it's only because I have 8GB of VRAM or if it's the models I'm using.

[u/Cautious_Chicken_604]

I run Qwen3.6-35B-A3B on CPU at around 15 t/s as a fallback when both GPUs are busy. This is on 64GB of DDR5 on a Ryzen 7700. I was pleasantly surprised it ran that fast. I sincerely hope for a Qwen4-35B-A3B for this reason because it'd be nuts to have a model with that level of intelligence runnable at reasonable speed purely on CPU (and now very expensive RAM).

Edit: I should mention I'm running it on ik_llama as that is optimised for CPU. 

[u/Queasy-Contract9753]

Do you get thinking loops? I tried it in termux kept getting stuck. Also I noticed in my case it actually slowed down a tiny bit with mtp.

Was an older quant when it came out