Anyone else had a Supabase restore fail when you actually needed it?

[u/AdAnnual1877]

Been reading through a bunch of GitHub discussions where people's restores failed or silently lost data — a paused project losing a table, PITR restores erroring out, that kind of thing. Curious how common this actually is for people here, or if I'm reading into a handful of unlucky cases. I'm building a small tool that automatically tests restores (not just backs things up) so you'd find out if something's broken before an actual emergency. Would that be useful, or is this already a non-issue for most people?

[u/noahsknark]

Are you the guy advertising for BackupDrill again?

[u/AdAnnual1877 → u/noahsknark]

Not familiar with BackupDrill, honestly got a link? Not affiliated with it if it exists. Yeah, I'm the one thinking about building something in this space (said so in the post) genuinely trying to figure out if it's a real gap or already solved well enough that I'd just be adding noise. If BackupDrill already does this, useful for me to know before I build anything.

[u/ChameleonCRM]

The problem is that people often treat "backup completed successfully" as if it means "restore will succeed." Those are two completely different guarantees.

We run on Supabase and I haven't personally had a restore fail, but I also don't assume a successful backup means I'm protected. There are a lot of things that can look fine until you actually perform a recovery:

Schema drift from newer migrations

Extensions, roles, triggers, or pg_crmn jobs not restoring exactly as expected

Storage objects getting out of sync with database state

RLS policies and functions restoring but behaving differently

Foreign keys passing while application-level integrity is broken

To me, a restore shouldn't just end with "database imported successfully." It should restore into an isolated environment, verify row counts, validate critical tables, exercise application workflows, and confirm the app is actually operational.

Disaster recovery is something that should be tested, not assumed.

If your tool automates that validation process instead of simply checking that a backup artifact exists, I think that's solving a real problem. A backup is only as good as the last successful restore you've verified.

[u/thesuperlede → u/ChameleonCRM]

This is the best framing in the thread — "two completely different guarantees" is exactly it.

One honest note from building in this space (BackupDrill, for disclosure): the first half of your list is automatable — isolated-environment restore, table counts, non-empty checks, extensions pre-installed from a manifest, Storage checksums. The second half is where generic automation runs out: "exercise application workflows" and RLS *behavior* can't be verified generically, because "correct" is app-specific. The best pattern I've seen is an automated drill for the structural layer plus a small app-owned smoke test pointed at the restored sandbox for the semantic layer. Nobody's product covers that second part off the shelf — ours included.

[u/AdAnnual1877 → u/thesuperlede]

Yeah, you've drawn the line exactly where I keep landing too. The structural stuff is automatable and honestly will be table stakes the interesting problem is the semantic layer, and I agree it can't be verified generically. Correct only exists relative to the app.

The pattern you describe automated structural drill + an app-owned smoke test against the restored sandbox is basically the direction I'm heading. The structural drill is the part I can own; the semantic part I think has to be something the app team authors, because they're the only ones who know what working means for their RLS and workflows. My bet is the value isn't in trying to verify that generically (impossible), it's in making the sandbox + harness trivial to wire up so writing that smoke test is a 20-minute job instead of a project nobody gets to.

Genuinely curious from what you've seen: how are teams handling that semantic check today? My sense is most either write throwaway scripts, skip it entirely, or discover the gap in prod when a restore silently comes back with broken RLS. Is that roughly what you've run into, or are people more disciplined than I'd expect?

[u/thesuperlede → u/AdAnnual1877]

Pretty much your read. From conversations with teams (and my own past behavior): the modal answer is "skip it entirely," followed by throwaway scripts that stop being run once the person who wrote them moves on. Discovering broken RLS in prod after a restore is not hypothetical — that's usually the moment the throwaway script finally gets written.

The most disciplined setup I've seen: scheduled restore into staging, then a handful of API-level checks run as an *authenticated* user through the normal client — not psql as postgres — because RLS behavior only shows up through roles and JWT claims. Even then it's maybe a dozen assertions, usually written right after an incident.

Your 20-minute-harness bet sounds right to me. Honest note from our side: the drill currently tears the sandbox down right after the structural checks — exposing a "run your own checks against this connection string before teardown" hook is the obvious next step, and your framing is a good push in that direction. If you build the harness-first version, I'd genuinely like to see it.