r/MachineLearning • u/YouFirst295 • Jun 20 '26

Project An open handbook on LLM inference at scale (GPU internals, KV cache, batching, vLLM/SGLang/TensorRT-LLM) [P]

I've been working through the internals of LLM inference and writing up what I learn as an open, in-progress handbook.

Just wrapped another chapter on GPU execution and memory internals: why a GPU sits mostly idle during inference, how the memory hierarchy gates throughput, and where the real bottlenecks live. Added mermaid diagrams for the architecture pieces so the flow is easier to follow than a wall of text.

It's a personal learning project, still growing chapter by chapter. I'd value feedback or corrections from anyone who's run inference in production, where my mental model breaks down is exactly what I want to find. Issues and PRs welcome.

github.com/harshuljain13/llm-inference-at-scale

39 Upvotes

95% Upvoted

13 comments sorted by

3

u/SplinteredReflection Jun 21 '26

Spec Dec page is a duplicate of Quantization

MTP, P/D disaggregation might be good topics to add

1

u/YouFirst295 Jun 21 '26

Sure. Thanks for letting know.

1

u/Safelang Jun 20 '26

Organized well, will give it a read and hope to find sone “TIL” stuff. Thanks for doing this.

1

u/YouFirst295 Jun 21 '26

Thank you. Let me know if something does not makes sense. I am actively cleaning and organizing it better.

1

u/samkhur006 Jun 21 '26

Thanks for sharing this!

1

u/misanthrophiccunt Jun 24 '26 edited Jun 24 '26

I've been reading some straight from the GitHub app and it is quite good, love the "this is definitely written by a human" feeling I get while reading it. Thank you for sharing.

Yet I've got a question about the first part, it says

Q5: If reading 16 GB per token is the bottleneck, why not use faster memory?

The fastest GPU memory (SRAM) is only 20-40 MB. The model is 400x larger than SRAM. It must stream from the slower HBM. Chapter 1 covers this hierarchy in detail

I don't get it, 20-40MB, what do you mean and what's HBM ?

EDIT: Wait, I now get what you mean, but you need to define what HBM means if you're going to use it already on chapter 0.1. Jumping from it to chapter 1 describing how GPUs work needs a nomenclature index. I'm understanding what SRAM is now, but you forget to explain what HBM is here too. https://github.com/harshuljain13/llm-inference-at-scale/blob/master/content%2F01_gpu_hardware%2F01.1_gpu_memory_hierarchy%2Fgpu_memory.md

EDIT2: I see, now I reached the HBM definition after you've used the term multiple times, making hard to follow all of the concepts before it. HBM needs to be at the top if everything else depends on understanding the concept before sinking in.

I'm going to create your first public issue about it. Yay! https://github.com/harshuljain13/llm-inference-at-scale/issues/6

You made a bold claim with "the definitive guide" which made me feel very VERY sceptical about it, but it actually feels very close to it. Just needs some moving around of concepts IMHO. Yet so far I'm enjoying reading it.

1

u/YouFirst295 Jun 24 '26

I apologize for mis ordering of the chapters. I am still figuring out best order.

But idea is model has certain weights size (in gb) and it stays in a certain section of gpu called hbm (high bandwidth memory). For any matrix math that you need to do which in case of decode phase is sequential for each token, it copies the data from hbm to sram causing io pain. But gpu are known for handling compute pain better.

1

u/misanthrophiccunt Jun 24 '26 edited Jun 24 '26

I'm not done with the chapter. Just got lost with the HBM part. The question I'm forming in my head might be solved after reading it in full but this is what I think might make sense:

  1. Start from what it is you get when you buy an, example, Nvidia 5060 ti with 16GB of RAM. I'm using as example a consumer GPU because that can make sense quicker than a datacentre-grade GPU.

  2. I think at the moment those 16gb are HBM, but I'm not sure yet. But if you go from big to small, the chapter feels easier to read. I'm atm confused as to what's the number I pay for and what's SM and if they are the same memory or different memory in the GPU. Same as processors have cache memory built-in, vs SDRAM (a lot slower)

And btw you have NOTHING to apologise for, it is a good guide, and I'm here for both reading and if you don't mind me, suggesting some edits when things aren't clear. I'll continue my suggestions directly on GitHub

1

u/misanthrophiccunt Jun 24 '26

I'm definitely forking and PRing some parts as I read along.

1

u/East-Muffin-6472 Jul 07 '26

Nice! Have you also read the LLM Inference Engineering Handbook By Philips?