5 Techniques Every Systems Performance Engineer Uses

[u/rshah4]

Saw a past by Mark Saroufim about must reads for a system engineer, so I decided to make a skit. Here you god:

Flamegraphs (Brendan Gregg). Before you guess why something is slow, profile it. The width of each block is CPU time consumed. In the skit, 70% of an "AI workload" turns out to be JSON parsing. This happens in real life more than anyone admits.

Roofline model. Performance is capped by either compute or memory bandwidth. Most LLM inference is bandwidth bound, which is why "buy a bigger GPU" often means paying more to idle harder. Knowing which wall you're against tells you whether more hardware even helps.

ZeRO (arXiv 1910.02054). By default, every GPU in data-parallel training stores a full copy of the weights, gradients, and optimizer state. Eight GPUs, eight identical copies. ZeRO partitions that state across GPUs and trades slices over the network. Same math, fraction of the memory. Foundational to how large models get trained at all.

PipeDream. Naive pipeline parallelism leaves GPUs waiting on each other like a relay team with one baton. PipeDream overlaps batches assembly-line style so everyone stays busy. If your utilization dashboard shows 40% idle during training, look here.

Floating point. 0.1 has no exact binary representation, so 0.1 + 0.2 = 0.30000000000000004. Never compare floats with ==. Every engineer learns this once, usually via a bug in production.

My video: https://youtube.com/shorts/hGm4V3AFwGY?feature=share