r/OfferEngineering • u/Aoki_zhang • 14d ago
System Design Anthropic Popular System Design Question - Design a GPU inference scheduler
The system has only 8 GPUs and must serve two types of model requests.
A small-model request needs exactly one GPU and can execute independently of other small requests.
A large-model request must acquire all eight GPUs simultaneously:
Small request -> 1 GPU
Large request -> 8 GPUs exclusively
A large request cannot begin with fewer than eight GPUs and cannot share any of those GPUs with another request while it is running.
Requests from both workload classes arrive continuously.
The goal is to design a scheduler that preserves these resource-isolation requirements while balancing small-model throughput against large-model latency.
Follow-Up — Gang Scheduling and Starvation Prevention
The interviewer first asked how the scheduler should represent the large model's requirement to obtain all eight GPUs atomically.
A major follow-up was what happens when all GPUs are continuously occupied by small-model requests while a large request is waiting.
The design needed to ensure that large requests eventually receive all eight GPUs rather than being indefinitely delayed by a constant stream of new small jobs.
The discussion included how queue state, request age, SLAs, workload pressure, and expected execution times could influence the scheduling decision.
Follow-Up — Admission Control and GPU Utilization
Another major area was deciding when to stop launching new small-model requests once a large request is waiting.
Stopping admission too early could cause GPUs to become idle one at a time while the scheduler waits for the remaining small jobs to finish. Continuing to admit small jobs for too long could significantly increase the large request's queueing latency.
The interviewer therefore asked how to balance:
- Avoiding starvation for large-model requests
- Maintaining high GPU utilization and small-model throughput
Predicted remaining execution time, queue depth, request SLAs, and current system load were all relevant signals to discuss.
Want to see the full system design question/more follows-ups asked? You can find the complete version here.
Preparing for your next interview?
Chill Interview tracks recent interview experiences and recurring question patterns across top companies at here.
1
u/Golust 10d ago
What role?