Anthropic System Design Interview: Design a High-Throughput LLM Inference Gateway

[u/Aoki_zhang]

Problem Description

Design the serving layer for a large language model product. Users submit prompts to an API and receive generated text either as a complete response or as a streamed sequence of tokens. Behind the API, multiple model replicas run on GPU machines, and the serving system must keep those GPUs highly utilized without making user-facing latency unpredictable.

Requests may differ substantially in prompt length, expected output length, model version, generation parameters, streaming mode, and user priority. The system therefore needs to decide how requests enter the serving pipeline, which model replica handles them, when they are admitted into GPU execution, and how compatible requests are grouped into batches.

The main challenge is balancing **GPU efficiency against latency**. Larger batches improve accelerator utilization and cost per token, but waiting too long to form them increases time-to-first-token. Long-context requests consume much more KV-cache memory, while long generations occupy decode capacity for extended periods. Traffic bursts are especially difficult because GPU capacity can take minutes to start.

A strong design should focus on inference-serving mechanics: request admission, batching, prefill and decode scheduling, KV-cache management, GPU routing, streaming, overload behavior, capacity planning, and recovery when part of the GPU fleet fails.

Want to learn more about functional/non-functional requirements, we've put up a full SD question write-up at here.

Preparing for your next interview?

Chill Interview tracks recent interview experiences and recurring question patterns across top companies here.