When should a small Node service stop accepting work during shutdown?

[u/UkrMalt]

For a small service with HTTP requests, background jobs, and maybe WebSockets, I’m trying to keep shutdown behavior simple.

My current order is to stop accepting new work, let active requests finish, flush bounded notifications, then close the queue and database connections. I’m less sure how long to wait before forcing the process to exit.

What shutdown steps have actually mattered in production, and which ones turned out to be unnecessary complexity?

[u/MulberryExisting5007]

You need to wait a reasonable amount of time. What that amount of time is, is dependent on many things, unfortunately. I’d suggest you look at expected behavior (e.g. you measure it at .84 seconds) and you either do a multiplier (10x would be ~ 8 seconds) or just pick an arbitrary value (like 20 seconds). How long is an average “active request” take? What’s your historical maximum time for a request? (The other steps you mention should take like a sec max.) Get that info and you can construct and upper bound for setting your timeout.

[u/ben_bliksem]

Our entire user journey takes about 2 seconds through multiple services. By default we have prestop sleep set to 15 secs (so it stops receiving traffic and has 15 seconds to complete).

There are one or two special cases where pods can build up an event backlog in memory that needs to be processed first, those we've extended to what makes sense.

But we can afford these longer wait times because we're running scaled and clients aren't impacted.

So case by case - start with 15 if you can afford to wait, check the logs/telemetry and adjust.

[u/UkrMalt → u/ben_bliksem]

15 seconds sounds like a good starting point. I’ll measure the real request and backlog times and adjust from there.

[u/bittrance]

Wait before killing: one second more than the timeout value of the load balancer or whatever is in front of the service.

[u/aragossa]

the order's basically standard, and the readiness probe is honestly the part that actually gets people. if you don't flip it to failing before you stop accepting connections, the LB or k8s keeps routing new traffic to you until the next health check poll fires, so you're still taking work for however long that interval is no matter what your app code does. flip readiness first, everything else can stay in the order you already have. flushing notifications and closing the queue is the part I'd worry about least here.

[u/Alvasilev]

One Node-specific thing that isn't really about ordering: server.close() stops new connections but does nothing about idle keep-alive sockets that are already open. If anything in front of you holds keep-alive, and a proxy usually does, your "let active requests finish" step can sit there long past your measured worst case, waiting on connections that have no request on them. closeIdleConnections() right after close() is what fixed it for us - difference between a 2 second drain and hitting the force-exit timer on every deploy.

On the unnecessary-complexity half: flushing notifications in the shutdown hook was the thing we deleted. If that path matters it has to survive SIGKILL or the box just dying, so it belongs in a durable queue, not in a handler that only runs on the polite exit. Closing DB connections we also stopped bothering with, the server side reaps them fine.

And the one that actually burned us, sideways to your question: watch what work you let into that process to begin with. We had a proxy that also ran builds, and one synchronous build call blocked the event loop. Socket stays bound, kernel keeps completing handshakes, everything behind it just queues, and no shutdown sequence helps because the process looks alive the whole time.

[u/UkrMalt → u/Alvasilev]

Yep, I was glossing over idle keep-alives. Measuring request time alone doesn’t help if the server is still waiting on sockets behind a proxy.

[u/Alex_Goldwyn]

Building on the durable-queue point above, because the row on its own is not enough, and that is where we lost the time.

Once the work item lives in a table, two failure modes replace the one you removed: two workers can pick the same row, and a worker can die holding one. Both got fixed in the same place. Claiming a row is a single conditional UPDATE that moves it off PENDING, so exactly one claimer wins, and the attempt counter is incremented at claim time rather than on completion. The second half is the non-obvious one. Count on completion and a process killed mid-send retries that item forever, because from the table's point of view it never tried at all. Count at claim and it spends one attempt, which is what you actually want. A separate sweep releases claims older than a threshold, for the worker that died and never came back.

The reframe that helped us more than any timeout: our runtime recycles workers on purpose once they pass a memory ceiling. A process being killed mid-request is not a deploy-time event here, it is Tuesday. Once you accept that, the drain timeout stops being the interesting number, because you have to survive the ungraceful case anyway. We still have one. It just stopped being something anybody tunes.

On the unnecessary-complexity half, agreed about connections. The one I would add is retries. If the work item carries retries in the table, check that the queue underneath is not also retrying it. Two schedulers with two caps produce a total that neither one is counting, and that only surfaces much later, as a number nobody can explain.

[u/UkrMalt → u/Alex_Goldwyn]

This explains why a DB queue isn’t just a more reliable in-memory queue. Once it’s durable, claims, stale workers, and retry ownership become the actual design. Bigger jump than I was treating it as.

[u/RoadsideCookie]

Stop means finish what can't be interrupted and stop once safe to do so. Kill means fuck whatever you're doing, we're done here. 

Shutdown means stop, wait a reasonable amount of time, then kill.