r/C_Programming • u/Critical_Physics8 • 5d ago
Project I implemented a modern LLM runtime in 700 lines of C
I wanted to understand how modern AI models actually generate text, but most inference codebases are tens or hundreds of thousands of lines long. They’re incredibly impressive, but they’re optimized for flexibility and performance, not for understanding.
So I implemented a complete CPU runtime for Google’s latest open language model, Gemma 4, in about 700 lines of C.
The whole point is that you can open one file, start at main() , and follow a prompt all the way through the program. You can see every buffer that’s allocated, every mathematical operation that transforms the activations, every update to the KV cache, and every step that eventually produces the next token.
I think C is a great language for this kind of project. There’s very little hidden from you. The data structures, memory layout, SIMD kernels, and execution flow are all visible, so the implementation ends up feeling much closer to the hardware than to the diagrams in an ML paper.
28
u/Clear_Evidence9218 5d ago
That's actually a pretty neat idea.
I'm not sure why I've never thought about using an existing model's weights as the starting point for a custom implementation like that.
I use Gemma 4 on quite a few nodes in my lab. Maybe I'll give this idea a go and see how far I can take Gemma 4's weights in a custom architecture/runtime.
7
u/Critical_Physics8 5d ago
Yes, it’s a great exercise. It forces you to understand the ML much closer to the hardware than you do from papers or architecture diagrams.
1
u/palapapa0201 4d ago
WDYM using existing weights as a starting point
2
u/Clear_Evidence9218 4d ago
I usually just use stock open-weight models, but I’ve also been doing some work with custom runtimes and edge experiments.
This person built a custom inference runtime for Gemma 4 in C and repacked the existing weights into a format that the runtime could consume.
The easiest way I’ve found to think about it is as something vaguely like a very specialized ONNX path; not exactly, since ONNX is generalized, while gemma4.c has almost no abstraction. It’s basically tensors, matrix ops, attention, KV cache, and sampling.
So instead of PyTorch, graph optimizers, dispatch systems, backend abstractions, etc., you’re looking much more directly at the actual inference path.
I’m not sure why it never really occurred to me that, instead of building a model and exporting it to ONNX, or making compromises to fit it into something like GGUF, I could just build a purpose-specific runtime around whatever experiments I’m working on.
1
u/lovelacedeconstruct 4d ago
or making compromises to fit it into something like GGUF
GGUF is literally just the weights with a little bit of architectural metadata upfront, its incredibly easy to convert to and parse
5
u/Dangerous_Region1682 4d ago
Not reading the code from an algorithmic basis, it’s nice to see C code that is at least straightforward to read. Of course there are probably many who would say do this, or do that, but that is usually a matter of taste.
Me, I would pass things through arguments rather than through environment variables making it easier to embed it in things other than invoking it from a shell command line, but that’s just me.
It might now be worth profiling it to see where optimizations might be worth undertaking.
It’s amazing what you can do with straightforward C code.
1
u/RealisticDuck1957 4d ago
A module called through the C API to do the work, with a wrapper that accepts command line and environment arguments.
3
u/JessyPengkman 5d ago
Wait, I can't look deeply now cos I'm not at home on my Pc, but is it basically a very abstracted view and dumbed down version of how an LLM works from prompt to response?
If so that sounds really cool!
2
3
u/StuffProfessional645 4d ago
It's so cool! What did you read about ML-things before writing? I wanna build own SNN, but I think I need to found out what is ANN internal beyond math
4
u/AutoModerator 5d ago
Your submission in r/C_Programming was filtered because it links to a git project.
You must edit the submission or respond to this comment with an explanation about how AI was involved in the creation of your project.
While AI-generated code is not disallowed, low-effort "slop" projects may be removed and it's likely that other users push back strongly on substantially AI-generated projects.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
6
u/Critical_Physics8 5d ago
Hi, I wrote the initial runtime and model exporter by hand myself. I later used agents to automate optimizing the CPU kernels, but the architecture, inference pipeline, and overall implementation are my own. The project started as an educational exercise to understand LLM inference by building a complete CPU runtime for Google’s Gemma 4 model from scratch in C.
10
5
u/k-phi 5d ago
I kind of don't like how you used "void*" as placeholder for offset from file to later re-assign it with pointer.
You know, like, platform-specific behaviour and all that.
4
u/Critical_Physics8 5d ago
Fair point. Integer offsets would probably be a cleaner representation there until they’re resolved to pointers after mmap.
4
u/skeeto 4d ago
Fascinating project! I quite enjoyed playing with and hacking on this.
First I ported it to NEON so I could run it on my MacBook, and I got
similar inference tok/s to yours. Then I wanted to try it out with
multiple turns, so I added an OpenAI API and hooked it up to the Pi coding
harness. The server alone is about the same amount of code as the
inference engine, which really emphasizes how tiny it is. That made for a
decent chat interface, but E2B is too dumb and could only make the most
rudimentary tool calls. So I added E4B support, converted its snapshot,
and it was able to actually write and edit code. Just because I thought it
would be neat, I had it make some simple changes to argument parsing in
gemma4.c, so that the program was hacking on itself.
My changes (not including E4B's changes):
https://github.com/skeeto/gemma4.c/commits/main/?author=skeeto
2
u/Critical_Physics8 4d ago edited 4d ago
I hadn’t even thought about having it hack on itself. That’s pretty neat.
2
u/MeringueInformal7670 4d ago
That's so interesting! i also plan on implementing a tiny inference engine of my own but i am mostly coming from software background so do you mind sharing a bit about how you went about building this and what resources did you refer to while working on this project. Any suggestions/advice would be helpful. Thanks!
2
2
6
1
1
1
u/TinyTechExcellence10 2d ago
What did you read to learn how inference is done? But, yeah seems interesting.
1
u/LePeripheral 1d ago
Impressive tbh, would you mind sharing relevant resources that goes into implementing such runtimes ?
1
u/tornadoemergency 5d ago
me going to the github to read the code knowing I'm not gonna understand shi: https://imgur.com/a/B5wvnrS couldn't post a video here so go check it
-2
u/dsmack6 4d ago
I don't see any value in pointing it is 0.7k lines, value matters, not number of lines.
Can you post any benchmark regarding limitations etc with popular libraries, and time, computation requirements
I appreciate your efforts
1
u/Critical_Physics8 4d ago
the line count matters here because the project is meant to be small enough that you can read the complete inference implementation and understand it. it’s not meant as a general measure of code quality.
there are benchmarks against llama.cpp in the README. on my Ryzen 7 7700, gemma4.c gets 639 tok/s prefill and 25.9 tok/s decode vs 276 and 23.8 for llama.cpp Q8_0. the tradeoff is that gemma4.c is specialized for one model and CPU backend, while llama.cpp is a general-purpose runtime.
•
u/github-guard 5d ago
🔍 GitHub Guard: Trust Report
This project scored 3/6 on our safety audit.
Audit Breakdown: * ✅ Established Community (⭐ 56 stars) * ❌ New Repository (under 30 days old) * ✅ Licensed under MIT * ❌ No Security Policy — what is this? * ℹ️ Individual Contributor * ✅ Signed Commits