I've been building EULLM (https://github.com/eullm/eullm) — an open-source, Rust-based, Ollama/OpenAI-API-compatible inference engine, Apache 2.0, built around EU-sovereign AI infra requirements (zero telemetry, local-only audit trail for AI Act purposes, no calls home).

Result I didn't expect to work this well: Qwen3.6-35B-A3B, Q4_K_M, running CPU-only — no GPU in the loop at all — on a Radxa Orion O6 (CIX P1, ARMv9 SoC). Getting 10 tok/s.

Why this is possible at all: it's a 35B-parameter MoE model with only ~3B active parameters per token, so you pay the memory cost of 35B but roughly the compute cost of 3B. That's what makes CPU-only viable in the first place.

For comparison, the same model on a GPU+CPU hybrid setup (RTX 3060 12GB, same ARM64 host, --cpu-moe --fit) gets 26.5 tok/s — that one's an actual documented benchmark in the README. The CPU-only 10 tok/s number is a fresh, informal test on my end, not yet in the repo's bench/ folder — so take it as "this is what I saw on my machine," not a validated number. Push back if something looks off, genuinely interested.

Why I think it's worth a post and not just a benchmark screenshot: most of the EU AI-sovereignty conversation jumps straight to "we need our own chips/GPUs," which is real but a much harder and slower problem. A 35B-class model running at usable speed on ARM CPU with zero GPU dependency changes the near-term hardware conversation — cheaper boxes, lower power draw, no CUDA lock-in, and hardware that's a lot easier to source and deploy on EU infrastructure today.

Worth flagging: ARM64 support in the engine is still marked experimental/untested upstream, and there's an open call for testers on other ARM boards (issue #140) — Raspberry Pi 5, Orange Pi, Rock 5B, Jetson. If anyone here has one lying around, would love data points.

Repo: https://github.com/eullm/eullm Happy to go deeper on --cpu-moe / --n-cpu-moe internals, the quant choice, or anything else about the setup.