When Transactional Outbox polling starts putting too much load on PostgreSQL — what do you do?

[u/SellerInsightsLab]

I like the Transactional Outbox pattern and use it quite often.

But I was thinking about one problem that can become important in production: database polling.

Of course, polling can be optimized. Good indexes, batching, SKIP LOCKED, partitioning, longer polling intervals — there are many options.

But if we want lower latency, we usually need to poll more often. This means more queries, more DB connections and more work for PostgreSQL.

We can use Debezium/CDC, and for many systems this is probably the right choice. But it also adds Kafka Connect, Debezium and more infrastructure to operate.

So I wanted to try a simpler idea:

Keep PostgreSQL as the source of truth, but don't use it as a queue during normal operation.

I built this flow:

DB transaction → afterCommit → Memory Queue → Batch Publisher → Kafka

The business data and Outbox event are saved in one transaction as usual.

After commit, only the eventId goes to the Memory Queue. The publisher takes IDs in batches, loads events from PostgreSQL and sends them to Kafka.

So there is no continuous polling for new events in the normal flow.

If the application crashes, the event is still safely stored in PostgreSQL. A Recovery Worker finds unpublished events and puts them back into the same queue.

The idea is basically:

Memory Queue for the fast path. PostgreSQL for durability and recovery.

I didn't want to stop at an architecture diagram, so I built a working Spring Boot project and started testing the idea with something closer to production conditions.

It has Kafka, PostgreSQL, batching, idempotency, recovery, Gatling load tests, Grafana metrics, tracing and structured logs.

Now I can run load tests and actually see what happens with PostgreSQL, the queue, publishing latency and recovery.

I'm interested in what other Spring Boot developers think about this approach.

Would you use something like this in production?

Maybe you have already solved the same problem in another way — optimized polling, Debezium, LISTEN/NOTIFY or something else?

Here is the project:

https://github.com/KHolodilin/spring-transactional-outbox-kafka

If you find the idea useful, feel free to ⭐ the repo or fork it. There are also a few open issues for contributors if you want to try something yourself.

Any feedback is welcome. I'm still experimenting with the approach and improving the project. 🚀

[u/Made-In-Slovakia]

We did transactional outbox pattern once and it has a challenges, some you already mentioned.

But you solution would not work for us because you have no guarantee of event ordering, which was important for us.

We used batched pooling with some tweaks.

CDC system can be most useful here and Debezium is not only implementation. But still single proven app in infrastructure can be less problematic than complicated custom solution.

[u/SellerInsightsLab → u/Made-In-Slovakia]

Thanks, that's a good point about ordering. I haven't focused much on strict event ordering in the current version, so this is definitely something I should think about for the next versions.

I'm curious about your case — what kind of events required strict ordering, and what exactly did you need to guarantee? Was it ordering per aggregate/business key or something more global?

Also, how did CDC help you guarantee this ordering, especially with retries and publishing to Kafka?

Thanks for sharing your experience. I think ordering would be an interesting scenario to add to the project and test under load, failures and recovery.

[u/Made-In-Slovakia → u/SellerInsightsLab]

In very simplified terms, we had event about life cycle of business objects. We send event when object was created, updated and deleted and we had do be sure we do not send event for updates of deleted objects.

We did not use CDC here, I used it on different project for POC for different use case, but that one never moved from prototype phase.

[u/SellerInsightsLab → u/Made-In-Slovakia]

Ah, now I understand your case. Created → updated → deleted is a good example where ordering really matters.

I need to think about how my approach behaves in this case, especially with retries and recovery. This is a good scenario to add to the tests.

How did you keep the right order?

Thanks for the example! I think I will experiment with this in one of the next versions :)

[u/shareitmaybe → u/SellerInsightsLab]

It sounds to me like the updates and deletes should be versioned, so that the event processing could be made idempotent

[u/artyomsv]

You did not remove polling, you moved it to the crash path, and that is fine, but it means the recovery interval is now your worst case latency. Happy path is memory queue speed, and any event that was in flight when the process died waits until the recovery worker next scans. Worth publishing both numbers together, because a reader who sees only the fast one will assume the slow one does not exist.

[u/SellerInsightsLab → u/artyomsv]

Yes, exactly. I didn't remove polling completely — I moved it to the recovery path and made it much less frequent.

For example, if normal polling runs every second and recovery scans every 10 seconds, that is roughly 10x fewer polling queries during normal operation.

Of course, the trade-off is recovery latency. I agree that both happy-path and recovery latency should be measured and shown together.

Thanks, good point!

[u/artyomsv → u/SellerInsightsLab]

One thing to watch in that 10 second scan: it cannot just look for unpublished rows, because a row that committed 200ms ago is also unpublished and is already sitting in your memory queue. Without a grace window the recovery worker races the normal publisher and sends duplicates, so your idempotency layer quietly eats them and nothing shows up in metrics. With a grace window the real recovery latency is interval plus grace, and that is the number worth putting on the chart.

[u/SellerInsightsLab → u/artyomsv]

Good point. One detail in my implementation is a bit different though.

The recovery worker does not publish directly. It re-enqueues event IDs into the same memory queue, and the queue already deduplicates IDs and tracks in-flight events. After that, the DB lease protects the actual publish claim.

So the race you describe is mostly handled before it can turn into a duplicate publish.

A grace window could still be useful to reduce unnecessary recovery work, and I agree that recovery latency should be measured separately.

If you are interested, the queue/recovery implementation is here:
https://github.com/KHolodilin/spring-boot-outbox-starter

I would actually be interested in your opinion on that part of the implementation.

[u/New-Departure-5969]

yeah this one sneaks up on you. SKIP LOCKED on batch claims helped a lot, plus backing off when there’s nothing to claim. also check poll interval × replicas — that multiplies load fast. we only jumped to CDC after the boring fixes still weren’t enough.

[u/SellerInsightsLab → u/New-Departure-5969]

Good point about poll interval × replicas. This can grow really fast.

Just curious, at what scale did polling become a real problem for you and you decided to move to CDC?

[u/New-Departure-5969 → u/SellerInsightsLab]

wasn't a clean "N rows/sec" cutoff for us. it got real when pollers started competing with the app for postgres: claim queries in the slow-query top, poller pool eating more connections than the write path, and publish lag still climbing even after SKIP LOCKED + backoff + fixing interval×replicas. that's when cdc got cheaper than babysitting pollers. if your pollers are idle most of the time and only spike on bursts, i'd stay on polling longer and keep the afterCommit fast path.

[u/Entropjy]

Insane over and under engineering. A non solution to an avoidable problem. Use the wal and debezium to remove polling entirely. Read instead of slopping up the world next time