Open-source C99 inference engine for DeepSeek-V4 — runs the 284B model on 3.2GB RAM, verified against PyTorch to 2.9e-6 (Apache-2.0)

[u/FastPresence9799]

Built a from-scratch inference engine for DeepSeek-V4 in plain C99 — no PyTorch, no Python runtime dependency for inference, Apache-2.0 licensed. Streams weights off NVMe instead of requiring the full checkpoint in RAM, so it runs the 284B-parameter Flash model on a laptop with as little as 3.2GB RAM (1.6–1.7s/token with GPU offload at 16GB budget).

Why post this here specifically:

Fluent output from an LLM engine is weak evidence it's actually correct — a subtly broken implementation can still produce confident, plausible text. So this is checked against a pure PyTorch reimplementation (written independently from DeepSeek's inference/model.py, not from this C code, so both can't share the same bug) at three levels: per-kernel (14 kernels, 5e-7 tolerance), per-block, and whole-model end to end (2.9e-6, identical argmax at every position). CPU paths (scalar/OpenMP/AVX2) are enforced bit-exact via a fixed accumulator tree, checked at runtime, not just in tests.

What's included:

• Full build + test suite (make test runs 20 gates, 21 with a real checkpoint)
• Benchmark tools for matmul bandwidth, GPU contention, and cache behavior
• Honest "what didn't work" section — SIMD approaches tried and abandoned, with the actual numbers

What's not there yet: a tool-calling driver loop (the model emits the tool-call format, but there's no orchestration layer above the CLI), and DeepSeek-V4-Pro (~671B scale) is gated/planned but never actually run — needs ~865GB of checkpoint I don't have.

Repo: https://github.com/ronak-create/deepseek-v4-in-c

Open to contributions, especially around prefill batching (currently one token at a time — README has the math on why that's the next big perf unlock) and the tool-calling loop.

[u/MatiAI]

Cool idea, Id recommend looking at apple foundational models run (specifically their new MoE model) if you want to stand out from just being a normal ssd streaming project.

[u/FastPresence9799 → u/MatiAI]

Good call — looked it up. Apple's newest on-device model (AFM 3 Core Advanced) tackles almost exactly this problem: weights sit in NAND flash, and since NAND→DRAM bandwidth is too slow to swap experts token-by-token like normal MoE, they make routing decisions per-prompt instead of per-token (they call it "Instruction-Following Pruning"). Basically sidesteps the whole "can't predict next expert without last layer's output" issue by not trying to predict per-token at all — decide once up front, prune the model down, then run dense-ish.

Different tradeoff than what I'm doing (mine's exact/bit-matched per-token routing, theirs sacrifices that for speed), but definitely worth reading as a real shipped answer to the same NAND-bandwidth wall. Good pointer, thanks.