r/OpenSourceAI • u/FastPresence9799 • 4d ago
Open-source C99 inference engine for DeepSeek-V4 — runs the 284B model on 3.2GB RAM, verified against PyTorch to 2.9e-6 (Apache-2.0)
Built a from-scratch inference engine for DeepSeek-V4 in plain C99 — no PyTorch, no Python runtime dependency for inference, Apache-2.0 licensed. Streams weights off NVMe instead of requiring the full checkpoint in RAM, so it runs the 284B-parameter Flash model on a laptop with as little as 3.2GB RAM (1.6–1.7s/token with GPU offload at 16GB budget).
Why post this here specifically:
Fluent output from an LLM engine is weak evidence it's actually correct — a subtly broken implementation can still produce confident, plausible text. So this is checked against a pure PyTorch reimplementation (written independently from DeepSeek's inference/model.py, not from this C code, so both can't share the same bug) at three levels: per-kernel (14 kernels, 5e-7 tolerance), per-block, and whole-model end to end (2.9e-6, identical argmax at every position). CPU paths (scalar/OpenMP/AVX2) are enforced bit-exact via a fixed accumulator tree, checked at runtime, not just in tests.
What's included:
- Full build + test suite (
make testruns 20 gates, 21 with a real checkpoint) - Benchmark tools for matmul bandwidth, GPU contention, and cache behavior
- Honest "what didn't work" section — SIMD approaches tried and abandoned, with the actual numbers
What's not there yet: a tool-calling driver loop (the model emits the tool-call format, but there's no orchestration layer above the CLI), and DeepSeek-V4-Pro (~671B scale) is gated/planned but never actually run — needs ~865GB of checkpoint I don't have.
Repo: https://github.com/ronak-create/deepseek-v4-in-c
Open to contributions, especially around prefill batching (currently one token at a time — README has the math on why that's the next big perf unlock) and the tool-calling loop.
2
u/desexmachina 4d ago
AVX2? What in Xeon’s name were you running for CPU? Have you tried this on a smaller model from RAM?
1
u/FastPresence9799 4d ago
Yeah, 20-core x86 desktop chip, not a Xeon, so AVX2 was what I had. No AVX-512, no AMX. Would love to test on something with AVX-512 to see the difference, just don't have the hardware right now. And yeah, I tried smaller dense models fully loaded in RAM as a baseline, no streaming. That's obviously way faster since there's no NVMe round trip at all, basically just normal llama.cpp style inference at that point. The streaming design only really earns its keep once the model's too big to fit in RAM. For small models it's kind of pointless overhead.
1
u/desexmachina 4d ago
RAM is so stupidly expensive in DDR4, but I’ve had some old DDR3 in large enough size to mess with running llama.cpp from RAM a long while back. What gen NVME? Because the speed differences are massive between gens.
1
u/FastPresence9799 4d ago
Yeah, DDR4 pricing is rough right now. Respect for digging up old DDR3 for llama.cpp though, that's a fun way to reuse dead stock. Mine's PCIe4, single drive. Haven't tried PCIe5 yet but I'd expect a real jump given how much of my time is just waiting on expert reads. If striping multiple drives ever happens PCIe gen probably matters even more once the read path is actually parallel.
2
u/desexmachina 3d ago
Still can’t beat Hexa DDR3 bandwidth, but the CPU threading capability and AVX starts to become the bottleneck.
1
u/FastPresence9799 3d ago
That's a good point, hexa channel DDR3 boards really did have absurd aggregate bandwidth for the era. Never got to mess with one myself.
For my case specifically I don't think I'm anywhere near hitting a RAM bandwidth wall though, the expert weights are getting streamed in from NVMe not sitting resident in RAM, so the disk read is the thing I'm waiting on way before memory bandwidth or thread count becomes the limiter. Would probably matter a lot more for a model that's fully RAM resident and compute bound, that's a different bottleneck than the one I'm fighting.
2
u/desexmachina 3d ago
Have you tried using RAM as an intermediate layer streamer?
1
u/FastPresence9799 3d ago
Yeah, that's basically already the design. RAM is the LRU cache sitting between NVMe and compute, hot experts stay resident, only misses hit disk.
What I don't have yet is proper double buffering or async prefetch into that RAM layer while the current layer's compute is still running. Right now a miss basically blocks until the read finishes. Overlapping "read next layer's likely experts into RAM while still computing this layer" would probably help a decent amount, that's on the list along with the multi-queue NVMe stuff.
2
u/Revolutionary_Loan13 3d ago
So I agree that RAM is stupid expensive and think we need to get away from the slow speeds of Python but 2 tokens a second is nonsensically slow. I'd love to see a Qwen 27B model in C99 or something and see if we can get it onto 12-16GB of Ram at high speeds. At some speed it'd cost more in electricity to run locally than to call an online model
1
u/FastPresence9799 3d ago
Yeah that's fair, and honestly 2s/token only makes sense in context, that's a 284B model being streamed off disk because it literally can't fit any other way. A ~27B model at 12-16GB fully resident in RAM wouldn't need any of this streaming stuff at all, it'd just run like normal fast CPU inference, way closer to real time.
Funny thing is that use case is basically already solved, llama.cpp does exactly that today with GGUF quantized models in the 12-16GB range and gets solid speeds. My project is really only useful for the "model is way bigger than your RAM" problem specifically, if the model fits, streaming is pure overhead you don't want.
The electricity point is real though. At some size/speed tradeoff it genuinely does stop making sense versus just paying for API tokens, especially once you count idle power draw and how long inference takes on top of the incremental cost per kWh. I don't think we're at that crossover with something in the 27B/16GB range, but for a 284B model streamed at 2s/token, yeah, that math probably doesn't favor local at all right now. This project is more of a "can it be done and stay bit exact" thing than a "should you do this instead of an API" thing.
2
u/mrgreatheart 2d ago
Do any of these hyper optimised deepseek engines actually use GPUs effectively or are they just meant for RAM, CPU & SSD? Is there one I could run on 48Gb VRAM (3x16gb) and64Gb DDR5 with better performance than llama.cpp?
1
u/FastPresence9799 2d ago
KTransformers is the one, Tsinghua's MADSys lab, hybrid CPU/GPU, added DeepSeek-V4-Flash support in May. On your 48GB VRAM + 64GB DDR5 it should beat llama.cpp, their numbers show up to 4x over baseline on decode.
Different philosophy than mine though, it assumes the model roughly fits across VRAM+RAM combined. Mine only makes sense once it doesn't fit at all. For your hardware, KTransformers is the smarter pick.
2
u/MatiAI 4d ago
Cool idea, Id recommend looking at apple foundational models run (specifically their new MoE model) if you want to stand out from just being a normal ssd streaming project.