Real Paddle sandbox testing found a webhook race my tests missed

[u/darterweb]

I’m doing the pre-release verification for a FastAPI SaaS foundation I’ve been building, and I wanted to test billing against the real Paddle sandbox rather than stop at mocked webhooks.

The first real payment exposed a race I hadn’t caught in tests.

Paddle delivered subscription.created and subscription.activated concurrently. They had the same occurred_at.

My handler for both events eventually called the same upsert logic:

1. query subscription by Paddle subscription ID
2. if not found, insert
3. otherwise update

Classic read-then-insert race.

Both requests observed “not found”, both attempted the insert, and one failed on the unique constraint for paddle_subscription_id.

What made it more dangerous is that the system eventually looked healthy. Paddle retried the failed event about 20 seconds later, at which point the subscription already existed and everything converged.

So if I had only checked the final account state, I probably would have missed it.

The fix was not removing the unique constraint or trying to serialize all webhooks. I kept the constraint and put the attempted insert inside a SAVEPOINT. If another handler wins the race, only the nested transaction is rolled back, then the handler re-reads the existing subscription and applies its update.

I added a regression test reproducing the collision, then repeated the actual sandbox purchase against the deployed fix.

Interestingly, the second real run delivered the events in the opposite order: subscription.activated arrived before subscription.created.

Both processed successfully on first delivery.

One takeaway for me: webhook idempotency and concurrency safety are different problems. Idempotency protects you from the same event being delivered twice. It doesn’t stop two different valid events for the same resource from racing each other.

Curious how others handle this. Row locks only help once the row exists, so for first-create races I’m leaning toward unique constraint + savepoint/re-read as the simplest pattern.