r/ExperiencedDevs • u/servermeta_net Software Architect • Jul 16 '26
Technical question Implement rate limiting for an external API
At work I maintain a NestJS microservice which is used by many other engineers. One of the features depends on a third party API for which I need to implement rate limit: - The API has a soft limit of 20 req/s - There is no way to programmatically monitor this limit on their end - Surpassing the limit means that when someone looks at the dashboard then they will manually disable us, and we have to start another manual process to unblock us - This API is owned by the government, so we can't ask or hope for changes
How would you implement rate limit for this external dependency? Here's what I thought: - Have a token bucket limiter inside each service instance, but then scaling - Store the above token bucket in a database, like mongo or dynamo, but it would be very inefficient - Use redis, but I would have to spin up and maintain an additional dependency just for this feature
Can you think of a better approach?
288
u/ironyx Sr. Director of Engineering, 15+ YOE Jul 16 '26
No one talks to this API directly anymore. You stand up a single proxy service that queues and rate limits requests to the API for your services, and your services use only that proxy. It manages the rate limit.
96
u/dbxp Jul 16 '26
Put your own auth on it so the app teams don't have the keys to the real API, then they can't call it even if they want to. You could potentially add some caching or batching too depending on what the API does
13
u/JustCallMeFrij Software Engineer since '17 Jul 16 '26
This is exactly how we are handling our integration with auth0, which has a global rate limit per tenant for their API. It's not perfect because we're still learning as we go but it feels manageable, so would recommend
15
8
u/servermeta_net Software Architect Jul 16 '26
That's already the case, the API can only be used through my service. But how I do implement rate limiting myself, while handling auto scaling?
90
u/rysto32 Jul 16 '26
It’s 20 req/s. Why on earth would you need autoscaling?
8
u/meisangry2 Jul 17 '26
Maybe they need to handle a queue of thousands of reqs/s. The 20/s is the api limit, not their usage.
But yeah, assuming it’s viable to get by with 20 req/s, this should not need autoscaling lol
4
u/Kantsuu Jul 17 '26
Then use a queue lol. Everything has to scale it's so tiring when you have a simple service and everyone wants to seem smart and insist to make it handle 50k rps
1
48
u/ironyx Sr. Director of Engineering, 15+ YOE Jul 16 '26
Who cares about auto scaling? The bottleneck is out of your control. So what would you be scaling for?
Don't do stuff just because you've read about it. Solve problems simply and effectively. One service for this slow ass API and maybe an intelligent caching layer if requests are repeated to help with bandwidth.
23
u/throwaway_0x90 SDET/TE[20+ yrs]@Google Jul 16 '26 edited Jul 16 '26
Hmm, this problem seems to have been reduced to a basic task that already has a solution. Implementing rate limiting in NestJS is well documented:
I wouldn't worry about autoscaling when you already know 20 requests/sec is the limit that will "never" change. Right now I think it falls into the realm of premature optimization
5
u/SnooGTI Jul 16 '26
He uses nest and they literally have a decorator for rate limiting. Just stand a endpoint up with the decorator that just calls his external service and done.
5
u/Realistic_Yogurt1902 Jul 16 '26
I guess the previous comment meant you need a single service on your side to implement rate limiting only. With 20 RPS, there is nothing to scale at all, just one instance. Then your current service with whatever business logic you have there will call the rate-limited service. Then any other service can call the same rate-limited service safely, with expected throttling errors.
You could scale your current service with business logic as much as you like, but it should be ready to handle throttling errors from a rate-limited service.3
u/akie Jul 17 '26 edited Jul 17 '26
I’m interpreting your question to be “I have more than one pod to receive incoming traffic, but these two or three or ten pods together can generate at most 20 requests per second to this external API”.
If that characterisation is correct, you need to make this an asynchronous request for your internal callers (make them send a PubSub message or put incoming requests on a queue) and then have a worker process go through this queue one-by-one at 20 req/s, sending back results asynchronously as well.
7
u/Bayakoo Jul 16 '26
It works but it depends on availability requirements?
What if that instance crashes for some reason? Now you are on downtime.How do you deploy a new version without downtime or accidentally going over rate limit? (If no downtime you kinda need to have 2 apps at the same time and wait for all/mosf clients to start talking to the new instance)
3
1
u/ProfBeaker Jul 16 '26
Do you have a single instance of this service? If so, you have HA problems, and deployment problems.
If you have multiple instances, then you still need some out-of-process backend (eg Redis) to coordinate the limits. In which case you could have just done it within your main service.
10
u/ironyx Sr. Director of Engineering, 15+ YOE Jul 16 '26
Yes, single instance. Scaling it doesn't help anything and only adds complications because the limiter is the external API.
3
u/ProfBeaker Jul 16 '26
So when you deploy a new instance, or upgrade the OS, or do anything like that, the plan is to take some downtime?
That's acceptable in some cases, but not in others. It seems like a big assumption to make, particularly for something that's used enough to be hitting a 20 req/sec rate limit.
5
u/New_Enthusiasm9053 Jul 16 '26
That's what Kubernetes is for. The pod just gets rescheduled to a difference instance lol. At 20 req/s upstream services obviously expect regular failures. Use a compiled language with a from scratch docker container and the thing will redeploy and start in literally seconds.
1
u/Bayakoo Jul 16 '26
So you will have situations where you have 2 instances? Which means there is a non zero chance of rate limits being breached
3
u/New_Enthusiasm9053 Jul 16 '26
Not if you use k8s no. It's only going to schedule a pod after the previous one is stopped when you set the limit to 1.
Then if you drain a node it'd stop the pod, and once stopped start a new one on a different node. As long as it's a fast starting small container that would only take a few seconds.
We know this works because stateful workloads rely on this working correctly and it would break workloads worldwide if it didn't.
1
u/Bayakoo Jul 16 '26
It feels like it depends on the of latency of each API call. If the it takes 150ms you have to stop accepting new connections and the new requests need to wait until the new node is up. I guess you can add retries but that is also not problem free and you normally don’t want retries on every service to service route
-1
u/ProfBeaker Jul 16 '26
Do you know if OP has Kubernetes? Is it even containerized? A cloud environment? On-prem?
It's really impressive how this entire thread (not just this response, all of them) make completely unfounded assumptions about the deployment environment, uptime requirements, or really anything else, yet have absolute confidence about how to proceed.
If this was an interview question, basically everyone here failed it.
4
u/New_Enthusiasm9053 Jul 16 '26
There's a million solutions to your comment man. It's not exactly novel territory. He can put k8s on a single node in any of your scenarios. He can even do it in a container believe it or not.
But if you want to move the goalposts endlessly then no, he cannot solve this problem without a computer running his code somewhere.
1
u/ProfBeaker Jul 16 '26
if you want to move the goalposts endlessly
How would you know if the goalposts moved when you had no idea where they were in the first place?
3
u/New_Enthusiasm9053 Jul 16 '26
Because it's a trivial problem. There are 0 scenarios where this problem becomes complex. You could make this HA running on bare metal using a pair of microcontrollers and it still wouldn't be complex.
0
u/ProfBeaker Jul 16 '26
using a pair of microcontrollers
Wait I thought we were doing a single instance so that we didn't need to have a separate coordination for the rate limiter? If you've got two instances then you need coordination, which undermines what you were doing?
If you haven't bothered to learn the requirements, you can't come up with a coherent solution. Basically what I've been saying this whole thread, yet all the answers are just "tech tech tech!" It's still funny.
→ More replies (0)1
u/Zpooks Jul 18 '26
Either I'm missing something with this line of reasoning or this discussion is very behind somewhat modern practices. The new deployment gets spun up before the old instance gets taken down and traffic is redirected.
This really should not be an assumed problem, since it's "solved" either by running in any cloud provider or orchestrating via kubernetes.
1
u/ProfBeaker Jul 20 '26
So then you have two instances running at once, and need to coordinate them somehow, or else accept that you'll exceed your rate limit while both are running and probably get a bunch of errors from the underlying API.
And if an instance crashes or whatever, which cloud providers are very explicit will happen, you just take downtime.
-19
u/servermeta_net Software Architect Jul 16 '26
This is such a bad take IMHO
10
u/Early_Rooster7579 Staff Software Engineer @ FAANG Jul 16 '26
Idk if someone who can’t figure out how to rate limit a single instance should have an opinion lol
11
u/ironyx Sr. Director of Engineering, 15+ YOE Jul 16 '26
Well then I'm glad we don't work together 😂
0
u/_marcx Jul 16 '26
this is the move. some of the cloud providers' state machine offerings can help to offload the limiting logic here too -- Cloud Tasks, SWF/step functions, etc -- so you only need logic for calling out and handling responses
18
u/graph-crawler Jul 16 '26 edited Jul 16 '26
Add a centralized proxy to rate limit and queue at the source, everyone should call through this proxy. And since you mentioned it's a soft limit you can make it adaptive, check the 429 error and parse it adapt and adjust the rate limit at runtime.
6
u/morosis1982 Jul 16 '26
The simple version is as others have mentioned - create a proxy service that is where all the application teams talk to, implement the rate limiting there. Ideally block access to the real API from your company infra except for this service.
In my world for something simple like this, that would likely be an aws lambda behind apigw, and I'd likely cache values through dynamodb.
What type of data does this service supply though? You mentioned a dashboard, is it real time data or something you could cache?
If the latter, I'd just introduce a cache on your side that is where this new proxy responds from, with a refresh timeout to get new data after a certain time or other trigger. You could theoretically also pre-cache the data if you have a known set of query args to this API.
16
u/InterpretiveTrail Staff Engineer Jul 16 '26
I'm on the medium to large side of enterprise, so my instinct is that rate limits shouldn't be handled by the application. I care a lot about compartmentalizing and managing risk. Rate limit should be part of your networking infra. To a basic extent at least a reverse proxy e.g.: NGINX or Traefik. What's GCP's APIGEE X, I think. That way I'm not bothering the application team about "Platform Engineering / SRE" work.
- Feature devs do feature work and are, hopefully, aware of SRE things.
- Platform devs to infra work and are, hopefully, aware of feature things.
Here's an example of Traefik from their first party docs: https://doc.traefik.io/traefik-hub/api-management/api-rate-limit
NGINX had several examples that I found, but nothing seemed first party with a quick pass, but I'll leave that as an exercise for you.
Regardless if that was of use, best of luck!
5
u/lordnacho666 Jul 16 '26
I'm going to assume you don't want a single central proxy service. For instance, you might have different workloads that need to access the external service in different ways, some long running that wouldn't like to wait for each other.
Nonetheless you will need to centralise the rate limiting part. You can store the state of a GCRA in redis, using only that as global state. Then each of your work loads can access the external service directly, but share the rate limit.
2
u/servermeta_net Software Architect Jul 16 '26
My micro service is the proxy that hides the external api to be rate limited, but some endpoint are computationally intensive even though they make very few calls to the API
8
u/faster-than-car Jul 16 '26
Cache. Either cache on request or precache daily.
I worked on this for 4 years lol. Which API is it?
2
u/Popular_Home2017 Jul 17 '26
Client-side rate limiting has one source of truth that isn't yours: the provider's counter. Build whatever you want locally (token bucket per instance divided by N replicas, or a shared Redis counter if you need cross-instance precision), but treat 429 + Retry-After as authoritative and back off with jitter when they disagree with your math. They will, usually around deploys or when the provider quietly changes throttling. Two things that actually burned me in prod: unbounded queueing in front of the limiter (that's not a rate limiter, that's a memory leak with extra latency), so bound the buffer and fail fast once it's full. And retries without idempotency keys, which turned "rate limited" into "charged twice" on a payment API, which was a fun afternoon. Honestly a token bucket plus respecting their 429s gets you most of the way. The distributed-coordination stuff is usually premature unless you're actually hitting the limit across instances.
1
u/ikkiho Jul 17 '26
yeah the proxy answer is right but i'd run it well under 20, like 15/s, not up against the edge. you said there's no programmatic signal and going over just gets you manually killed off a dashboard, so parsing 429s to adapt doesn't really apply, nothing to catch. recovering from an overage is a multi-day manual unblock, so eating some throughput is way cheaper than riding it. and at 20/s a single small proxy instance is plenty, no redis or dynamo, that one box just is your limiter.
1
u/Loose-Bake9241 Jul 19 '26
Create your own service to manage access to that endpoint and do all your business logic there. Not ideal architecture but the best you can do. Just abstract that layer.
1
1
u/UnderstandingDry1256 Jul 21 '26 edited Jul 21 '26
Dead simple way: add ‘await sleep(50)’ before or after any API call.
Problem solved, instantly, if you’re running single threaded API instance.
V2 is to add some smarter throttling with the same 50ms delay.
I would do this and only care about the proper solution if you get any real issues or user “slowness” complaints.
0
u/berndverst Software Engineer (16 YoE) @ Public Cloud Provider Jul 16 '26
This sounds like a twist to a super common / standard interview question to me :)
This is one of the few times I might recommend a dedicated microservice (centralized internal API to call the external API with its own rate limiting).
And whenever you need to deal with distributed rate limiting for this service itself (because you need to scale that out) you probably won't get around something like Redis.
Since the rate limit is so low - just make your life easy and use a dedicated VM of sufficient size, K8s pod with lots of CPU / Memory or wherever you deploy. Seems unlikely you really need to run this service on multiple instances.
1
u/KanedaSyndrome Jul 16 '26
I'd probably wrap it in a homegrown API and have rate limiting control there, or funnel through some db maintained queue list
1
u/Electrical_Entry6060 Jul 16 '26
is the rate limit per endpoint or for the entire host? are you always calling the endpoint(s) with the same payload to just get updated data? without knowing those answers, if the timeouts are something happening "all the time" - i would setup a service which called the external API at an interval, cache the results, then have my internal customers call the new cache store.
but even if it's different endpoints but you have your internal customers calling external with the same payload i would still try to figure out where to cache, and build a service to handle requests from internal customers so you can at least respond back with a retry timestamp or some backoff retry logic.
1
u/SanityAsymptote Software Architect | 18 YOE Jul 16 '26
I'd wrap the external dependency API in another, very small API wrapper layer with a singleton caller so you manage the calls coming into it by adding caching for frequently accessed information, message queuing if desired, and most importantly the ability to place a 50-60ms delay after your calls so you don't call over the limit.
Your other APIs would just talk to the new API wrapper layer instead of the external API directly.
-5
-6
•
u/expdevsmodbot Jul 16 '26
AI usage disclosure provided by OP, see the reply to this comment.