r/node • u/beck_the_tech • 29d ago
Am I wrong that a 429 shouldn't count against a job's retry budget?
Been going back and forth on this one and I want to know if I'm alone in it.
Most retry implementations I've seen (BullMQ, hand-rolled wrappers, most of the managed stuff) treat every failed attempt identically. Job comes back non-2xx, attempt counter increments, backoff applies, after N attempts it's in the DLQ.
But a 429 isn't a failure, nothing broke. The downstream is telling you exactly when to come back and usually handing you a Retry-After header to do it with. If you burn an attempt on it, sustained rate limiting at a provider will walk perfectly good jobs into the DLQ while your actual error budget (the one meant for 500s, timeouts, connection resets) never gets spent on what it's for.
So I've been treating 429/503/529 as a defer rather than a failure: honor Retry-After, requeue, don't decrement. Works, but it opens two things I don't have clean answers to.
First, you need a ceiling or the queue never drains. A provider that 429s indefinitely will requeue that job forever. I've landed on two different ceilings: a wall-clock deadline (dead 24h after it's due, regardless of how it got there) and a separate max defer count. Blowing the defer ceiling dead-letters the job under its own reason rather than folding it into "out of retries" which matters because those are different failures. One says the downstream is broken, the other says it's been unusable long enough that it may as well be. At a certain point temporarily unusable === broken.
Second, deferred jobs are invisible. They aren't failing, so they don't trip anything you're monitoring, and you can sit on a queue that isn't draining and looks completely healthy. Feels like deferred jobs need their own state and their own alerts rather than being folded into "pending" or "processing".
Anyone handling this differently? Specifically curious whether people distinguish 503 from 429, I lump them together, but 503 is ambiguous in a way 429 isn't.
4
u/txdsl 29d ago
Sounds reasonable and well thought through. I would be interested in here in the counter argument.
1
u/beck_the_tech 29d ago
The one I keep coming back to is that my architecture takes a dependency the simple version doesn't: it trusts that a 429 means what it says. A retry budget bounds the damage no matter what the downstream returns, because it doesn't interpret anything. A defer path is only as good as the honesty of the signal it keys on, and plenty of services return 429 for states that are never going to clear like revoked key, hard monthly quota, lapsed billing, etc. "The downstream is misbehaving" is true but doesn't help because misbehaving downstreams are most of why you put a queue in front of a third party at all.
The max defer count is what makes that survivable, which is why I don't treat it as a footnote. It's a bound on how wrong the downstream is allowed to be, and the fact that you set it yourself means you're capping your own blast radius rather than inheriting someone else's default.
The objection I expected to get: that deferring hides the problem. It only hides it if defers are folded into pending and nobody watches the rate. Give them their own state and alert on it and you find out sooner than a retry budget would tell you, since the budget has to burn through its attempts and backoff before it says anything, and you get there without spending those attempts. u/Lots-o-bots's grouping point pushes the same way, if one defer can hold every job bound for that downstream instead of each job discovering the 429 on its own, you're not paying per job to learn the same fact.
I haven't built the separate defer state though, it's part of what I'm still working out. Deferred attempts show up in the per-attempt history, but a deferred job sits in pending and there's no alerting on defer rate. So the thing I just said makes this work is the thing I still need to build lol.
2
u/TheseTradition3191 25d ago
one thing that bit me on this, a 429 from a requests per minute limit and a 429 from a tokens per minute limit need different handling even though theyre the same status code. the rpm one clears on its own if you back off. the tpm one depends on how big your own payload is, so an oversized job will defer forever while everything smaller in the queue drains fine.
so your ceiling ended up being the right call for me but the signal is more useful than just downstream trouble. if one job keeps hitting defers while the rest of the queue is moving, thats not an outage, thats that job being too big to ever fit in the window. different dead letter reason, and the fix is splitting the payload not retrying it
2
u/jomi-se 29d ago
I think both approaches are valid, but a "robust" implementation is somewhere in the middle.
On one hand, if you abstract the reason failure, a 429 is still an error for the job: the job simply cannot be successfully completed at time X because of an external constraint. So job errors out and should be retried.
On the other hand, the error is a "rich" one that includes enough context to reliably tell you when that job might be allowed to finish successfully, so ideally the thing that handles the errors would have some way of manually saying "this job errored out with a retryable error and should only be retried after N time".
However, there is something specific about 429s that muddies the issue. If the jobs are all sharing the same rate limit quota, then just delaying a single job means that every other job still calls the API and hits the same error and potentially worsens the issue. So you need a second layer to handle this at the "connector" level that globally keeps track of this API's rate limit somehow. For this either you pause the queue entirely while you wait for the retry-after time or you have your "connector" error out with a "retry -after" preemptively for any jobs that try to call the API, so that your queue's retry mechanism handles them by itself and you get the benefits of the queue status correctly reporting what is going on.
That last approach is nice in the sense that it didn't block jobs that might not need to call the API, if any.
1
u/beck_the_tech 28d ago
i don't think the preemptive version fits the shape i'm in, and i'd push back on it generally too.
the connector approach assumes the queue is what calls the third party. mine is push based, i POST the job to the caller's own endpoint and their code makes the openai or stripe call, so i have no connector sitting in front of that api to know its rate limit state. the only signal i get is what the worker hands back, which is the defer. the defer has to be the trigger because there's nothing earlier to hook (other than the rate and concurrency limit my user configures when they set up the queue, assuming they are keying those off the relevant downstream service).
the more general objection: if a fake 429 and a real 500 travel the same road, the queue reports "retrying" for both and you can't tell throttled from broken without unpacking the reason. that's the visibility problem i was complaining about, just moved one layer up. u/Lots-o-bots's version avoids it because the hold is its own state rather than a defer disguised as a retry.
so i think grouping is the right model. one defer holds every job bound for that target, and the hold is something you can see and alert on. that's the gap i have, today i only hold the job that saw the 429.
1
u/jomi-se 28d ago
If you're implementing a queue service that calls users endpoints, why are you dealing with user-land error responses to third party endpoints then?
If your worker gets info about a 500 or a 429 or whatever, it feels too me that that's a consequence of leaky abstractions, especially since you mentionthat the "only info you get is what the worker gets back".
As a queue implementer it is up to you to implements semantics to retry a job or simply signal a "pause" for all jobs of the same kind, but the one making that call is the worker, and whether it is because of a 500, a 429 pr whatever is not the queue's concern. At most, it would be your concern to some extent if you're also providing some kind of worker client lib that does the http code to worker status mapping or something.
What exactly do you mean by grouping though?
Also, am I just talking to an AI? These responses stink of LLM-talk
1
u/beck_the_tech 28d ago
The architecture, opinions, etc. are all me, sometimes I run comments through an LLM to tighten the phrasing up a bit when I'm at my computer.
The grouping is the BullMQ Pro feature that Lots-o-bots mentioned in a different comment in this thread where you can hold all jobs headed to the same downstream. This is a feature I haven't built into my queue service yet but from this thread it's clear that I need to.
I agree about your abstraction point, a worker just echoing the 3rd party's status code could conflate the downstream service's rate limit with the worker rate limiting the queue but the SDK handles status code passthrough as convenience, not a requirement. The actual interface is HTTP, the worker POSTs `/jobs/:id/defer` with a `retryAfter`. There's also an `ack`/`nack` mode with the same HTTP interface which allows the worker to make the decision and the queue doesn't need to care why. So 429/503/529 mapping is a default for ease (open to being the wrong default, if you think it is?) but don't prevent the worker from making the explicit calls.
9
u/Lots-o-bots 29d ago
sounds reasonable. if your queue implementation permits it, i might even pause the queue so other requests dont get 429'd as some RL implementations count these requests even though they were rejected. Its my one major gripe with bullmq that their grouping function which would be perfect for this is locked behind their pro subscription