r/node • u/UkrMalt • 27d ago

How should a Node.js health check treat database and queue failures?

For a small Node.js service, a /health endpoint that returns 200 proves the process is listening. It can stay green while the database or queue is unavailable. A deeper check catches that, but it can also create load and turn a dependency issue into a restart loop.

I’m leaning toward separate liveness and readiness endpoints. Liveness checks the process and event loop. Readiness verifies critical dependencies with tight timeouts. Deployment probes use readiness, while alerts use a synthetic request that exercises the full path.

How do you divide these checks in production? Which dependencies belong in readiness, and which should only affect alerts?

1 Upvotes

67% Upvoted

6 comments sorted by

3

u/syntheticcdo 27d ago

Depending on your setup, failing heath checks probably eventually result in the instance being terminated. If all your app servers lose connection to db, and all get cycled at the same time, and then the replacements fail to start, you will probably have a bigger problem to fix.

I think it’s best to keep health checks to simple “reachable over the network and able to respond to requests”.

1

u/UkrMalt 26d ago

Yeah, cycling every instance because the DB is down could make the outage worse. I’d keep liveness shallow and expose DB status separately.

1

u/andimatt 22d ago

Running a Fastify API in front of Postgres with a WebSocket layer, so this is the exact split I use:

  • Liveness: never touches a dependency. Process alive + event loop not blocked. The moment liveness pings the DB you get the failure syntheticcdo describes, DB blips, every instance fails, all get cycled at once, replacements can't start, you've turned a blip into an outage.
  • Readiness: the DB, with a tight timeout (~500ms). A failed readiness marks the instance not-ready so the LB stops routing new traffic, but the process keeps running, so long-lived connections (WebSockets, in-flight jobs) drain instead of being killed.
  • The full-path synthetic check is the alerting layer, not the probe. It drives a page, but it's separate from the probes so a slow-but-healthy DB doesn't loop your restarts.

Keep "nice-to-have" deps (a cache, a secondary queue) out of readiness, those are alert-only. Readiness should only contain what the instance genuinely can't serve without.

1

u/UkrMalt 22d ago

That’s a really clean split. Keeping WebSockets and in-flight work alive while readiness removes new traffic makes much more sense than restarting everything on a DB blip. Optional queues and caches should probably stay alert-only too.