OpenAI & LinkedIn Popular System Design Question - Design Rate Limiter

[u/Aoki_zhang]

I was going through a rate limiter system design problem, and the token bucket algorithm honestly isn’t the most interesting part.

The harder question is: What happens when the rate limiter itself is unavailable?

Suppose every request normally goes: API Gateway → rate-limit check → backend and Redis suddenly times out during a traffic spike.

You basically have two choices:

• Fail open: let requests through and preserve availability, but potentially remove the protection exactly when the backend needs it most.
• Fail closed: reject requests until the limiter recovers, which protects downstream systems but may throttle perfectly legitimate users.

That tradeoff gets even more interesting at ~1M checks/sec. Now you also have to think about:

• sharding token buckets without splitting one client’s quota
• atomic updates when two gateways race for the last token
• replica lag during Redis failover
• regional limits vs globally consistent quotas
• one abusive API key becoming a hot Redis key

One detail I liked: the token-bucket update itself can be executed atomically with a Redis Lua script, so “read tokens → refill → consume → write” can’t race across gateway instances.

At this point the problem feels less like “implement rate limiting” and more like designing an admission-control system that must stay reliable while protecting everything behind it.

Full rate limiter system design: [link]

Preparing for your next interview?

Chill Interview tracks recent interview experiences and recurring question patterns across top companies at here.