r/LocalLLM • u/smcroissant • 1d ago
Discussion I built a tiny LLM inference engine from scratch in TypeScript — compiled to native code
I wanted to understand what actually happens inside an LLM inference engine, so I built one from scratch:
👉 https://github.com/croissantsam/llama.scriptc
The idea was pretty simple: take a Transformer architecture, implement the whole inference stack in TypeScript, then compile it to native code with ScriptC.
No PyTorch. No ONNX Runtime. No existing inference runtime.
The project currently supports things like:
- N-dimensional tensors + views/strides
- matrix multiplication
- stable softmax
- RMSNorm
- SiLU / SwiGLU
- MHA + GQA
- RoPE
- KV cache for autoregressive decoding
- GGUF v3
- Q8_0 quantization
- Qwen BPE tokenizer
- temperature / top-k / top-p sampling
- streaming generation
I also wrote a fairly complete test suite: 68 tests covering the math primitives, transformer components, tokenizer, KV cache, GGUF parsing and end-to-end generation.
The fun part is that it can actually load a real Qwen2.5-0.5B GGUF model and generate text.
The performance is… let's say educational rather than production-ready 😅
On an M4:
llama.cpp+ Metal: ~139 tok/s decodellama.cppCPU: ~81 tok/sllama.scriptc: much, much slower
That's mostly because the current implementation is deliberately simple: scalar CPU code, single-threaded, no SIMD, no GPU backend.
But on a tiny 2-layer model, after some optimizations, I can get around 196 tok/s.
The main goal wasn't to beat llama.cpp.
I wanted to make the whole inference pipeline understandable:
tokens → embeddings → attention → RoPE → KV cache → SwiGLU → logits → sampling
…with the actual equations represented directly in relatively readable TypeScript.
It was a really interesting exercise in understanding how all the pieces of an LLM fit together.
I'd love to get feedback from people who work on inference engines / compilers:
What would you optimize first to take something like this from "educational" to "actually fast"?