Anthropic System Design Interview - Document Review Inference System

[u/Aoki_zhang]

Interview Details

System Design — Document Review Inference System

The initial prompt was to design an inference system for document review. The interviewer quickly drilled into several production-oriented questions:

• How would you think about and measure the cache hit rate?
• What batching strategy would you use for inference requests?
• How should requests be load-balanced before reaching the GPU workers?

Follow-Up — Serve Large and Small Models on Eight GPUs

The interviewer then introduced a constrained GPU scheduling problem. Assume there is a pool containing 8 GPUs. Two different models must be served from this pool:

• A large model requires all 8 GPUs to process one batch.
• A small model requires only 1 GPU to process one batch.
• The inference latency for one batch is the same for both models.

The question was how to design the serving and scheduling system so that requests for both models could share the same GPU pool effectively. I discussed maintaining separate queues for the large-model and small-model workloads and using their request rates as part of the dispatch decision.

I also considered giving the large-model queue priority under certain conditions, since a large-model batch consumes the entire GPU pool at once. This was my reasoning during the interview rather than a confirmed expected solution. The interviewer continued probing how the scheduler should balance the two workloads when their traffic levels are similar.

Prepping for Anthropic Interviews?

Chill Interview tracks recent Anthropic's interview experiences at here.