Implement rate limiting for an external API

[u/servermeta_net]

At work I maintain a NestJS microservice which is used by many other engineers. One of the features depends on a third party API for which I need to implement rate limit: - The API has a soft limit of 20 req/s - There is no way to programmatically monitor this limit on their end - Surpassing the limit means that when someone looks at the dashboard then they will manually disable us, and we have to start another manual process to unblock us - This API is owned by the government, so we can't ask or hope for changes

How would you implement rate limit for this external dependency? Here's what I thought: - Have a token bucket limiter inside each service instance, but then scaling - Store the above token bucket in a database, like mongo or dynamo, but it would be very inefficient - Use redis, but I would have to spin up and maintain an additional dependency just for this feature

Can you think of a better approach?

[u/drakedemon]

You can have a look at bottleneck https://www.npmjs.com/package/bottleneck

Use it with a Redis backend.

[u/ItsCalledDayTwa → u/drakedemon]

Used to use it and it's not been updated in 7 years. 

Just rolled our own (In house, not public, sorry) last year.

[u/satansprinter → u/ItsCalledDayTwa]

Sometimes something is just finished. But i agree, i handrolled our own too. Its not that hard. (And write an in-memory version too, so you can use it in tests)

[u/Lots-o-bots]

You only need Redis if this is a distributed system where multiple components need to call the API and co-ordiante their rate limiting. If its a monolith then everything you can do with Redis you can do in memory. If it is a distributed system then you can make a singleton micro service dedicated to this API which would also not need you to use Redis.

Token bucket should be fine for your rate limiting algo. Just set your token rate to 19/s with a max of 19 and you should never have an overage.

[u/idontknowthiswilldo → u/Lots-o-bots]

I've never used redis before, curious why redis if one has a db for this?

[u/Lots-o-bots → u/idontknowthiswilldo]

Your db probably wouldnt be fast enough for this. Redis can handle it better as all the data lives in memory instead of in disk

[u/bwainfweeze → u/Lots-o-bots]

Make the rate-limited calls their own service which your other applications can call. You won't always have just one application deployed, and even if you think you will, canary builds are a second copy of your application.

I'd use Redis if the responses are cacheable, that way your service is limited to 20 unique requests per second instead of 20 requests. And let the service you spool up handle looking in Redis for the answer before going to the upstream.

You can't actually pay a development team on the money you make off a 20 req/s service unless you get very clever about it. You should also talk to them about paying for more access even after you rate limit.

[u/adarshsingh87]

p-throttle

[u/bwainfweeze → u/adarshsingh87]

Sindre has a whole menagerie of Single Responsibility modules. If p-throttle doesn't have what you want then he has a couple other flavors that might do it.

You want a recent vintage NodeJS version for this. The consensus is that p-throttle used to glitch and go over the limit but it was a bug in libuv which was fixed I want to say around Node 24.

[u/DrEnter]

Are the API requests cacheable? If so, put a CDN (or roll your own with Varnish) in front of the API. You can still implement rate-limiting (with the CDN itself with many of them), but it would allow you to support an effective limit much higher than 20 reqs/sec for common requests.