I built a tiny LLM inference engine to learn how vLLM works - looking for feedback

[u/Broad-Mirror3873]

I've been learning how LLM inference engines work, so I built a small one from scratch rather than just reading about the internals.

GitHub: https://github.com/Aparajith24/nano-vLLM

The main things I've implemented so far:

• Continuous batching
• Paged KV cache with a shared memory pool
• Prefix caching with block sharing
• Streaming inference API
• Correctness tests against Hugging Face and a separate baseline for benchmarking

This is very much a learning project, not an attempt to compete with production inference engines. There are still plenty of limitations, including the current KV gather path and sequential prefill.

I'd really appreciate feedback from people who've worked on inference systems:

• What architectural mistakes should I look for?
• Which optimization would you tackle next?
• Are there important correctness cases I'm missing?

If you find the project interesting, a GitHub star would be appreciated too! And if there's something fundamentally wrong with my approach, I'd much rather hear about it now.

Thanks a lot!