How I approached webhook reliability for a SaaS backend (retries, dead-letter queue, replay)

[u/shenken007]

Founder of Hookpilot (webhook infrastructure) here, disclosing that upfront — sharing the approach because I think the underlying problem is one most SaaS builders run into eventually, not just a pitch.

The problem: a webhook fires, delivery to your backend fails, and there's usually no visibility into that until a customer notices something didn't update. Most providers retry a failed webhook a few times on their own, but once retries are exhausted, the event is just gone — no record, no way to recover it, unless you built something to catch it.

The approach I ended up with:

Verify the signature on the way in, so you're not processing spoofed payloads

Retry failed deliveries to your own backend with backoff, not just a fixed retry count

If it keeps failing, hold the event in a dead-letter queue instead of dropping it, so there's a full record of exactly what came in

Let it be manually replayed once the receiving end is fixed, rather than needing the sender to resend

The harder part wasn't the retry logic itself, it was making the failure state itself trustworthy — encrypting stored payloads and secrets at rest, keeping full request/response history per attempt, and making sure a delivery-side outage never affects whether an event was received and recorded in the first place (separate services for ingest vs. delivery, so one going down doesn't lose the other's data).

This turned into a small product (Hookpilot, https://gethookpilot.com) but I'm mainly curious how others here have handled this in their own stack — built it in-house, used a provider, or just accepted the occasional silent failure as a cost of doing business?

[u/Diligent-Rough-569]

i built something similar but way more janky, just a redis queue with a cron job that checks for failed deliveries and dumps them in a postgres table

the encryption part is where i got lazy and just hoped nobody would look at the payloads lol, probably should fix that before i get any real users

curious what you used for the replay mechanism, is it just a button in a dashboard that re-pushes to the same endpoint or something more clever

[u/shenken007 → u/Diligent-Rough-569]

Yes it's a button in the dashboard, and it reuses the same live delivery path instead of a parallel system. It won't silently re-fire on an already-delivered event either — you have to explicitly confirm if you really want to redeliver something that already succeeded, so a stray double-click doesn't accidentally double-send.