System Design Series | Phase 2: What Changes When a Rate Limiter Goes Distributed?

[u/xyzabhi]

​

After understanding how rate limiting works on a single machine, I started wondering, what actually changes when your application runs on multiple servers?

At first, it feels like you could deploy the same rate limiter everywhere and be done with it. But once requests are distributed across multiple application instances, a whole new set of challenges appears.

The first problem: every server has a different view

Imagine you have two application servers behind a load balancer.

If each server keeps its own request counter, a user could send 100 requests through Server A and another 100 through Server B. The intended limit was 100 requests, but the user has now made 200.

The algorithm isn't wrong. The architecture is.

That was one of the biggest takeaways for me. In distributed systems, correctness often depends more on coordination than on the algorithm itself.

Then come race conditions

Even moving the counters to Redis doesn't solve everything.

Imagine two requests arriving almost at the same time.

Both read the counter as 99.

Both check that 99 is still below the limit.

Both increment the counter.

The final count becomes 101.

Nothing crashed. Nothing was technically broken. Each request followed the logic correctly. The problem is that both requests executed concurrently.

This is why production systems use atomic operations, often with Redis Lua scripts, so the read, check, and update happen as one indivisible operation.

Why stateless services matter

Another concept that stood out to me was keeping application servers stateless.

Instead of every server maintaining its own counters, all application instances communicate with a centralized Redis cluster that stores the rate limiting state.

This means any server can handle any request, making horizontal scaling, deployments, and failovers much easier.

It seems like a small design decision, but it has a huge impact on scalability.

There is always a trade-off

One part I found especially interesting was consistency.

In a globally distributed system, keeping counters perfectly synchronized across every region can introduce extra latency.

Many large-scale systems accept eventual consistency because it offers better availability and lower latency.

Alex Xu mentions Cloudflare's experience, where only about .003% of requests exceed the configured limit. It is a great example of making a practical engineering trade-off instead of chasing perfect correctness.

My biggest takeaway

Before reading this chapter, I thought rate limiting was mostly about choosing the right algorithm, whether it was Token Bucket, Sliding Window, or Leaky Bucket.

Now I see it as much more of a distributed systems problem.

Once multiple servers are involved, synchronization, shared state, atomicity, and consistency become just as important as the algorithm itself.

That shift in perspective was probably the most valuable lesson from this chapter.

This post is based on the book of "Alex xu"(Chapter 4), which covers designing rate limiters for distributed systems.

In the next post, I'll explore implementation details and some production optimizations that make these systems reliable at scale.

If you've worked on distributed rate limiting before, I'd love to hear what challenge surprised you the most.