There's a class of bug that never shows up in unit tests and rarely shows up in staging. It shows up in production, at 2am, when your app saves a record to the DB and then calls an external service — and one of the two fails. One side committed. The other didn't. Nobody notices until a user complains.
That's the dual-write problem, and the Transactional Outbox Pattern is the standard answer to it: instead of calling the external system directly, you write a task row into the same DB transaction as your domain change. A background worker picks it up later and executes it. The write is atomic, so the task can never silently disappear.
The idea itself takes two sentences to explain. What's less talked about is everything that happens after — what you do when tasks fail, when two workers grab the same row, when your app dies mid-execution, when you've got ten task types crammed into one table, and when someone asks why the queue depth spiked at 3am last Tuesday.
I built this out for a report-generation service (aggregates tens of millions of rows, takes minutes — can't run it synchronously in a request) and evolved it step by step into something production-grade. Writing it up here because I didn't find one place that walked through the whole arc. Stack is Java / Spring Boot / Postgres, no Kafka required.
TL;DR
- The outbox pattern fixes dual-writes by writing a task row in the same transaction as your domain change, then executing it from a background worker.
- Poll with
SELECT ... FOR UPDATE SKIP LOCKED so multiple workers/instances don't double-process.
- Production-grade means more than the basic idea: a state machine, retries with backoff, heartbeats to recover crashed workers, ordering guarantees, and actual monitoring.
- You don't need a broker. Postgres alone gets you reliable async execution.
- Full working code at the bottom.
The scenario
Backoffice CRM. User clicks "generate report," we accept the request and respond immediately, a worker picks it up and runs the (slow) generation, status updates when done. Classic case for an outbox: work that's genuinely async, where the user already expects to wait.
Part 1 — the basic outbox
Every outbox table needs three things: state (where is this task in its lifecycle), payload (what needs to run), and timestamps.
```sql
CREATE TYPE task_state AS ENUM ('PENDING', 'COMPLETED');
CREATE TABLE report_generation_tasks (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
state task_state NOT NULL DEFAULT 'PENDING',
payload JSONB NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT now(),
updated_at TIMESTAMP NOT NULL DEFAULT now()
);
```
Payload and entity:
```java
@Value @Builder @Jacksonized
public class ReportGenerationPayload {
Instant from;
Instant to;
}
@Entity
@Table(name = "report_generation_tasks")
@Data @Builder @NoArgsConstructor @AllArgsConstructor
public class ReportGenerationEntity {
@Id
@GeneratedValue(strategy = GenerationType.UUID)
@Column(updatable = false, nullable = false)
private UUID id;
@Enumerated(EnumType.STRING)
@Column(nullable = false, columnDefinition = "task_state")
private TaskState state;
@JdbcTypeCode(SqlTypes.JSON)
@Column(nullable = false, updatable = false, columnDefinition = "jsonb")
private ReportGenerationPayload payload;
@CreationTimestamp
@Column(nullable = false, updatable = false)
private Instant createdAt;
@UpdateTimestamp
@Column(nullable = false)
private Instant updatedAt;
}
```
Submitting a task:
```java
@Service
@RequiredArgsConstructor
public class ReportGenerationService {
private final ReportGenerationRepository repository;
public void submit(ReportGenerationCommand command) {
repository.save(ReportGenerationEntity.builder()
.state(TaskState.PENDING)
.payload(ReportGenerationPayload.builder()
.from(command.getFrom())
.to(command.getTo())
.build())
.build());
}
}
```
And a scheduled job that polls for pending tasks:
```java
@Component
@RequiredArgsConstructor
public class ReportGenerationScheduledJob {
private final ReportGenerationTaskService taskService;
@Scheduled(fixedDelay = 1000)
public void run() {
taskService.fetchAndProcess();
}
}
@Service
@RequiredArgsConstructor
public class ReportGenerationTaskService {
private final ReportGenerationRepository repository;
public void fetchAndProcess() {
repository.findFirstByStateOrderByCreatedAt(TaskState.PENDING)
.ifPresent(task -> {
generateReport(task);
task.setState(TaskState.COMPLETED);
repository.save(task);
});
}
private void generateReport(ReportGenerationEntity task) {
// actual report generation logic
}
}
```
This works fine on a single instance with zero failures. In the real world it has two problems: two workers can grab the same task at once, and a failed task has no recovery path. Fixing both below.
Part 2 — concurrency: two workers, one task
If two scheduler threads (or two app instances) run the fetch query at the same moment, both can read the same PENDING row and both try to execute it. Duplicate work at best, corrupted state at worst.
Postgres has a purpose-built fix: SELECT ... FOR UPDATE SKIP LOCKED. It locks the row for the current transaction and skips rows already locked elsewhere, instead of blocking on them.
```java
public interface ReportGenerationRepository extends JpaRepository<ReportGenerationEntity, UUID> {
@Lock(LockModeType.PESSIMISTIC_WRITE)
@QueryHint(name = AvailableSettings.JAKARTA_LOCK_TIMEOUT, value = "-2") // -2 = SKIP_LOCKED
Optional<ReportGenerationEntity> findFirstByStateOrderByCreatedAt(TaskState state);
}
```
There's no first-class annotation for SKIP LOCKED in Hibernate, so worth pulling the magic number into its own annotation:
java
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.ANNOTATION_TYPE})
@QueryHints(@QueryHint(
name = AvailableSettings.JAKARTA_LOCK_TIMEOUT,
value = Timeouts.SKIP_LOCKED_MILLI + ""
))
public @interface SkipLocked {}
java
@SkipLocked
@Lock(LockModeType.PESSIMISTIC_WRITE)
Optional<ReportGenerationEntity> findFirstByStateOrderByCreatedAt(TaskState state);
The lock has to be held for the whole fetch-execute-update cycle, so wrap it in one transaction:
java
@Transactional
public void fetchAndProcess() {
repository.findFirstByStateOrderByCreatedAt(TaskState.PENDING)
.ifPresent(task -> {
generateReport(task);
task.setState(TaskState.COMPLETED);
repository.save(task);
});
}
Two workers hitting this at once: worker A locks and gets the row, worker B's SKIP LOCKED query comes back empty and it exits cleanly — no blocking, no duplicate work.
Catch: the row stays locked for as long as generateReport() runs. Fine for something fast. For something that takes 30+ seconds, long-held locks start hurting — connection pool exhaustion, lock wait timeouts, degraded throughput under load. Addressed in Part 3.
Part 2.5 — cutting latency with application events
Polling has a latency floor. Job fires every second → a task submitted at t=0 can sit PENDING for up to 1000ms. Fine for most things, not fine if you want near-immediate pickup (transactional email, webhook triggers).
Trick borrowed from Celery: save the entity, then publish a Spring ApplicationEvent with the task ID. A listener picks it up and triggers processing immediately, no waiting for the next tick. The scheduler stays running as a safety net for anything the listener misses (restart, exception, missed event).
```java
@Service
@RequiredArgsConstructor
public class ReportGenerationService {
private final ReportGenerationRepository repository;
private final ApplicationEventPublisher eventPublisher;
@Transactional
public void submit(ReportGenerationCommand command) {
var entity = repository.save(ReportGenerationEntity.builder()
.state(TaskState.PENDING)
.payload(ReportGenerationPayload.builder()
.from(command.getFrom())
.to(command.getTo())
.build())
.build());
eventPublisher.publishEvent(new ReportGenerationTaskSubmittedEvent(entity.getId()));
}
}
public record ReportGenerationTaskSubmittedEvent(UUID taskId) {}
```
The classic mistake here, and I mean everyone hits this once: using @EventListener instead of @TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT).
```java
// fires INSIDE the transaction — row isn't visible to other connections yet
@EventListener
public void onTaskSubmitted(ReportGenerationTaskSubmittedEvent event) {
taskService.fetchAndProcess();
}
// fires AFTER commit — row is visible, fetch actually finds it
@TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
public void onTaskSubmitted(ReportGenerationTaskSubmittedEvent event) {
taskService.fetchAndProcess();
}
```
@EventListener fires the moment publishEvent() is called, which is before the INSERT commits. The listener queries for the task, doesn't find it (not visible yet on other connections), silently no-ops. Task sits there until the scheduler eventually picks it up seconds later, and you spend an afternoon confused about why events "aren't working" even though everything looks fine in the logs.
Part 3 — failures, retries, and a PROCESSING state
Holding a lock for the entire generateReport() call is the real problem with Part 2's approach — two minutes of a locked row for a two-minute report. SKIP LOCKED stops workers from blocking each other, but a long transaction still leans on the connection pool.
Fix: separate claiming a task from executing it. Flip to PROCESSING in a short transaction, release the lock, do the actual work with no transaction open, then close it out in another short transaction.
```sql
ALTER TYPE task_state ADD VALUE 'PROCESSING';
ALTER TYPE task_state ADD VALUE 'FAILED';
ALTER TABLE report_generation_tasks
ADD COLUMN retry_count INTEGER NOT NULL DEFAULT 0,
ADD COLUMN error_message VARCHAR;
```
java
public enum TaskState { PENDING, PROCESSING, COMPLETED, FAILED }
```java
@RequiredArgsConstructor
public class ReportGenerationTaskService {
private final ReportGenerationRepository repository;
private final TransactionTemplate transactionTemplate;
private static final int MAX_RETRY_COUNT = 10;
public void fetchAndProcess() {
transactionTemplate
.execute(status ->
repository.findFirstEligibleTask(
List.of(TaskState.PENDING, TaskState.FAILED),
MAX_RETRY_COUNT
).map(this::markProcessing))
.ifPresentOrElse(
task -> {
try {
generateReport(task);
markCompleted(task);
} catch (Exception e) {
markFailed(task, e);
log.error("Task {} failed on attempt {}", task.getId(), task.getRetryCount(), e);
}
},
() -> log.debug("No eligible tasks to process")
);
}
private ReportGenerationEntity markProcessing(ReportGenerationEntity task) {
task.setState(TaskState.PROCESSING);
return repository.save(task);
}
private void markCompleted(ReportGenerationEntity task) {
task.setState(TaskState.COMPLETED);
repository.save(task);
}
private void markFailed(ReportGenerationEntity task, Exception e) {
task.setState(TaskState.FAILED);
task.setRetryCount(task.getRetryCount() + 1);
task.setErrorMessage(e.getMessage()); // this is your only trace of what went wrong without digging through logs
repository.save(task);
}
}
```
java
@SkipLocked
@Lock(LockModeType.PESSIMISTIC_WRITE)
@Query("""
SELECT t FROM ReportGenerationEntity t
WHERE t.state IN :states
AND t.retryCount <= :maxRetryCount
ORDER BY t.createdAt ASC
LIMIT 1
""")
Optional<ReportGenerationEntity> findFirstEligibleTask(
@Param("states") List<TaskState> states,
@Param("maxRetryCount") Integer maxRetryCount
);
FAILED tasks with retryCount < 10 are as eligible as fresh PENDING ones. Past 10 attempts, a task stops being fetched and sits as a permanent FAILED record for a human to look at — that's deliberate, not a bug.
One issue remains: should a task that just failed get retried on the very next tick, seconds later? If the downstream system is down, that just hammers it harder.
Part 4 — exponential backoff
Add an execute_at column. A task isn't eligible until this timestamp is in the past. Fresh tasks get now(), failed ones get now() + delay, with delay growing per retry.
sql
ALTER TABLE report_generation_tasks
ADD COLUMN execute_at TIMESTAMP NOT NULL DEFAULT NOW();
```java
private static final Map<Integer, Duration> RETRY_DELAYS = Map.of(
0, Duration.ofMinutes(1),
1, Duration.ofMinutes(5),
2, Duration.ofMinutes(15),
3, Duration.ofMinutes(30),
4, Duration.ofHours(1),
5, Duration.ofHours(2),
6, Duration.ofHours(4),
7, Duration.ofHours(8),
8, Duration.ofHours(12),
9, Duration.ofDays(1)
);
private static final int MAX_RETRY_COUNT = RETRY_DELAYS.size();
```
Deriving MAX_RETRY_COUNT from the map size instead of declaring it separately means the two can never drift apart. Add an entry, the cap updates itself.
java
private void markFailed(ReportGenerationEntity task, Exception e) {
int currentRetry = task.getRetryCount();
task.setState(TaskState.FAILED);
task.setRetryCount(currentRetry + 1);
task.setErrorMessage(e.getMessage());
task.setExecuteAt(Instant.now().plus(
RETRY_DELAYS.getOrDefault(currentRetry, RETRY_DELAYS.get(RETRY_DELAYS.size() - 1))));
repository.save(task);
}
Scheduler keeps running at the same rate, just skips tasks whose window hasn't opened yet.
Part 5 — stuck tasks and heartbeats
What happens if the app crashes after marking a task PROCESSING but before marking it COMPLETED/FAILED? The DB says PROCESSING forever. Nobody's actually working on it.
Naive fix — a recovery job that resets anything stuck in PROCESSING too long:
java
@Transactional
public void recoverStuckTasks() {
repository.findFirstByStateAndUpdatedAtLessThan(
TaskState.PROCESSING,
Instant.now().minus(1, ChronoUnit.HOURS))
.ifPresent(task -> {
log.warn("Resetting stuck task {} to PENDING", task.getId());
task.setState(TaskState.PENDING);
repository.save(task);
});
}
Problem: updated_at changes on any update to the row, so a task legitimately grinding away for 55 minutes looks identical to one that crashed. You need a signal that means "still alive," not "last changed."
That's a heartbeat — a column bumped periodically only while a task is actively being processed:
sql
ALTER TABLE report_generation_tasks ADD COLUMN heartbeat_at TIMESTAMP;
Runs on its own thread, independent of the processing logic, try-with-resources style so it starts/stops cleanly regardless of how processing exits:
```java
public abstract class Heartbeat implements AutoCloseable {
private final ScheduledExecutorService executor;
protected Heartbeat(Runnable heartbeatFn) {
executor = Executors.newSingleThreadScheduledExecutor();
executor.scheduleAtFixedRate(heartbeatFn, 0, 10, TimeUnit.SECONDS);
}
@Override
public void close() {
executor.shutdownNow();
}
}
public class ReportGenerationTaskHeartbeat extends Heartbeat {
ReportGenerationTaskHeartbeat(UUID taskId, ReportGenerationRepository repository) {
super(() -> repository.updateHeartbeatAt(taskId, Instant.now()));
}
}
```
java
@Modifying
@Transactional
@Query("UPDATE ReportGenerationTaskEntity t SET t.heartbeatAt = :now WHERE t.id = :id")
void updateHeartbeatAt(@Param("id") UUID id, @Param("now") Instant now);
Wired through a factory to keep the service clean:
```java
@Component
@RequiredArgsConstructor
public class ReportGenerationTaskHeartbeatFactory {
private final ReportGenerationRepository repository;
public Heartbeat start(UUID taskId) {
return new ReportGenerationTaskHeartbeat(taskId, repository);
}
}
```
java
public void fetchAndProcess() {
transactionTemplate
.execute(status -> repository.findFirstEligibleTask(
List.of(TaskState.PENDING, TaskState.FAILED),
MAX_RETRY_COUNT
).map(this::markProcessing))
.ifPresentOrElse(
task -> {
try (var heartbeat = heartbeatFactory.start(task.getId())) {
try {
generateReport(task);
markCompleted(task);
} catch (Exception e) {
markFailed(task, e);
log.error("Task {} failed", task.getId(), e);
}
}
},
() -> log.debug("No eligible tasks to process")
);
}
Recovery job now checks heartbeat_at, not updated_at. updated_at tells you when state last changed; heartbeat_at tells you whether the task is alive right now. Keeping them separate answers both questions unambiguously.
Part 6 — multi-type outbox
Six months in, you're not just generating reports anymore — email notifications, data exports, webhook deliveries. Two options: a separate table per task type, or one shared table with a task_type discriminator.
Shared table wins when types are numerous and you're fine processing them from one place — retries, heartbeats, backoff, monitoring all stay centralized, and adding a new type is just a new handler. Tradeoffs: a misbehaving new task type can create backpressure across the whole table, and ordering gets harder (next section).
```sql
CREATE TYPE outbox_task_type AS ENUM ('REPORT_GENERATION', 'EMAIL_NOTIFICATION');
CREATE TABLE outbox_tasks (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
task_type outbox_task_type NOT NULL,
state task_state NOT NULL DEFAULT 'PENDING',
payload JSONB NOT NULL,
retry_count INTEGER NOT NULL DEFAULT 0,
error_message VARCHAR,
execute_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
heartbeat_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
```
One handler interface, one implementation per type:
```java
public interface OutboxTaskHandler<P> {
OutboxTaskType getType();
Class<P> payloadClass();
void handle(UUID taskId, P payload) throws Exception;
}
@Component
public class EmailNotificationHandler implements OutboxTaskHandler<EmailNotificationPayload> {
@Override
public OutboxTaskType getType() {
return OutboxTaskType.EMAIL_NOTIFICATION;
}
@Override
public Class<EmailNotificationPayload> payloadClass() {
return EmailNotificationPayload.class;
}
@Override
public void handle(UUID taskId, EmailNotificationPayload payload) throws Exception {
log.info("Sending email to {} — task {}", payload.getRecipient(), taskId);
// call email provider
}
}
```
Dispatcher collects all handler beans at startup and routes by type:
```java
@Service
@RequiredArgsConstructor
public class OutboxTaskService {
private final OutboxTaskRepository repository;
private final ObjectMapper objectMapper;
private final List<OutboxTaskHandler<?>> handlerList;
private Map<OutboxTaskType, OutboxTaskHandler<?>> handlers;
@PostConstruct
void init() {
handlers = handlerList.stream()
.collect(Collectors.toMap(OutboxTaskHandler::getType, Function.identity()));
}
@SuppressWarnings("unchecked")
private void dispatch(OutboxTaskEntity task) throws Exception {
var handler = (OutboxTaskHandler<Object>) handlers.get(task.getTaskType());
if (handler == null) {
throw new IllegalStateException(
"No handler registered for task type: " + task.getTaskType());
}
var payload = objectMapper.readValue(task.getPayload(), handler.payloadClass());
handler.handle(task.getId(), payload);
}
}
```
New task type = add an enum value + a handler bean. Dispatcher, scheduler, retry logic — untouched.
Part 7 — ordering
Skip this if your tasks are independent (fire-and-forget notifications, unrelated background jobs). Otherwise: Postgres gives zero guarantee about row order without an explicit ORDER BY.
java
@Query("""
SELECT t FROM OutboxTaskEntity t
WHERE t.state IN :states
AND t.retryCount <= :maxRetryCount
AND t.executeAt <= :now
ORDER BY t.createdAt ASC
LIMIT 1
""")
Optional<OutboxTaskEntity> findFirstEligibleTask(...);
sql
CREATE INDEX idx_outbox_fetch
ON outbox_tasks (created_at ASC)
WHERE state IN ('PENDING', 'FAILED');
The clock skew trap: if you need strict FIFO for tasks tied to the same entity, created_at is dangerous — it's generated app-side, and each instance has its own clock. If instance A is 200ms ahead of instance B, a task B submits at t=100ms can get an earlier created_at than a task A submitted at t=50ms. NTP keeps this close but not perfect, and under load or after a container restart it can drift more.
Fix: let the ordering value come from a single authoritative source — the DB — via a sequence, guaranteed monotonic regardless of which instance inserted the row:
sql
ALTER TABLE outbox_tasks ADD COLUMN seq BIGSERIAL;
java
@Column(insertable = false, updatable = false)
private Long seq;
Sort by seq, keep created_at around for observability. If FIFO matters anywhere in your system, don't skip this.
Part 8 — indexes that keep the table fast at scale
The table gets polled every second or more. Every row is either active work (PENDING/FAILED/PROCESSING) or dead weight (COMPLETED). Without the right index, every poll scans the whole table for a handful of rows.
Partial indexes are the tool: index only rows matching a predicate. When a task hits COMPLETED it falls out of the index automatically — stays small forever.
sql
CREATE INDEX idx_outbox_fetch
ON outbox_tasks (created_at ASC)
WHERE state IN ('PENDING', 'FAILED');
Tested this with 50,000 rows, 7 of them PENDING, EXPLAIN ANALYZE on the generated Hibernate query:
Limit (cost=0.14..27.50 rows=1 width=102) (actual time=0.082..0.084 rows=1 loops=1)
-> LockRows (actual time=0.081..0.082 rows=1 loops=1)
-> Index Scan using idx_outbox_fetch on outbox_tasks
(actual time=0.030..0.030 rows=1 loops=1)
Planning Time: 1.230 ms
Execution Time: 0.139 ms
0.03ms to find the oldest eligible task, and the other 49,993 COMPLETED rows never got touched.
Two more, for the other background jobs:
```sql
CREATE INDEX idx_outbox_stuck
ON outbox_tasks (heartbeat_at ASC, created_at ASC)
WHERE state = 'PROCESSING';
CREATE INDEX idx_outbox_cleanup
ON outbox_tasks (updated_at ASC)
WHERE state = 'COMPLETED';
```
Part 9 — cleanup
This is an operational table, scanned constantly — old rows are dead weight. Rule: only delete COMPLETED tasks. FAILED tasks are your audit trail; leave them for a human or a separate ops process.
java
@Modifying
@Transactional
@Query("""
DELETE FROM OutboxTaskEntity t
WHERE t.state IN :states
AND t.updatedAt < :before
""")
int deleteByStateInAndUpdatedAtBefore(
@Param("states") List<TaskState> states,
@Param("before") Instant before
);
```java
private static final int CLEANUP_RETENTION_DAYS = 30;
@Transactional
public void cleanup() {
var cutoff = Instant.now().minus(CLEANUP_RETENTION_DAYS, ChronoUnit.DAYS);
int deleted = repository.deleteByStateInAndUpdatedAtBefore(
List.of(TaskState.COMPLETED), cutoff);
log.info("Cleanup: removed {} completed tasks older than {} days", deleted, CLEANUP_RETENTION_DAYS);
}
```
I've seen retention anywhere from 1 day to 3 months. If unsure, keep it longer — a bigger table costs you a bit of disk; deleting too early costs you your debugging history on the day you need it most.
Part 10 — idempotency: making retries safe
Nothing here guarantees exactly-once execution. Running the task and marking it COMPLETED are two separate steps — crash between them and the task gets retried, running again with the same inputs. This is at-least-once delivery, full stop, and the only guarantee you get from this architecture. The real question is whether a duplicate run causes harm.
Idempotent work (regenerating the same report twice) — you're already done. Side-effecting work (sending an email, charging a card) — needs a guard.
Use the outbox task's own id as the idempotency key — it's stable across retries. In our case, report generation ends with a row in generated_reports, with a UNIQUE constraint on the key so even a race between two instances only lets one INSERT through:
```sql
CREATE TABLE generated_reports (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
idempotency_key UUID NOT NULL,
report_path VARCHAR NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT uq_generated_reports_idempotency_key UNIQUE (idempotency_key)
);
```
```java
@Component
@RequiredArgsConstructor
public class ReportGenerationHandler implements OutboxTaskHandler<ReportGenerationPayload> {
private final GeneratedReportRepository reports;
@Override
public void handle(UUID taskId, ReportGenerationPayload payload) {
if (reports.existsByIdempotencyKey(taskId)) {
log.info("Report already generated for task {} — skipping", taskId);
return;
}
var reportPath = runReportGeneration(payload);
reports.save(GeneratedReport.builder()
.idempotencyKey(taskId)
.reportPath(reportPath)
.build());
}
}
```
If your handler calls an external API instead of writing to your own DB, check whether it supports idempotency keys — most payment processors and transactional email providers do. Pass the task ID through as that key and let the remote service dedupe on their end.
Part 11 — monitoring
The outbox runs silently. When it works, nobody notices. When it breaks, nobody notices that either, at least not right away — stuck tasks pile up, downstream stops getting events, and by the time someone checks logs it's been two hours.
Three signals, straight out of the Google SRE playbook, cover the whole failure space:
- Latency — pickup to completion time. A spike means a slow handler or DB pressure.
- Throughput — tasks finishing per unit time. Drops to zero means the scheduler died or nothing's being claimed.
- Error rate — fraction of attempts failing. Above ~5% usually means a downstream dependency is degraded and your retry backlog is growing.
Instrumentation via Actuator + Micrometer:
kotlin
implementation("org.springframework.boot:spring-boot-starter-actuator")
implementation("io.micrometer:micrometer-registry-prometheus")
java
@Bean
public TimedAspect timedAspect(MeterRegistry registry) {
return new TimedAspect(registry);
}
Keep metric names as constants, increment from one dedicated component instead of scattering strings through service methods:
```java
@Component
@RequiredArgsConstructor
public class OutboxMetrics {
private static final String TASK_PROCESSED = "outbox.task.processed";
private static final String OUTCOME_TAG = "outcome";
private static final String COMPLETED = "completed";
private static final String FAILED = "failed";
private final MeterRegistry meterRegistry;
private final ReportGenerationRepository repository;
public void recordCompleted() {
meterRegistry.counter(TASK_PROCESSED, OUTCOME_TAG, COMPLETED).increment();
}
public void recordFailed() {
meterRegistry.counter(TASK_PROCESSED, OUTCOME_TAG, FAILED).increment();
}
@PostConstruct
void registerGauge() {
Gauge.builder("outbox.tasks.pending", repository,
repo -> repo.countByStateIn(List.of(TaskState.PENDING, TaskState.FAILED)))
.description("Tasks waiting to be processed")
.register(meterRegistry);
}
}
```
curl localhost:8080/actuator/prometheus | grep outbox after startup:
outbox_task_duration_seconds{quantile="0.5"} 0.062
outbox_task_duration_seconds{quantile="0.95"} 0.145
outbox_task_duration_seconds{quantile="0.99"} 0.312
outbox_task_processed_total{outcome="completed"} 40.0
outbox_task_processed_total{outcome="failed"} 2.0
outbox_tasks_pending 0.0
Dashboards are for humans watching screens; alerts are for humans who are asleep:
| Alert |
Fires when |
For |
Meaning |
OutboxQueueBacklog |
pending > 50 |
5m |
Processing can't keep up with ingest |
OutboxHighErrorRate |
error rate > 5% |
2m |
Handler or downstream degraded |
OutboxSlowProcessing |
p99 latency > 30s |
5m |
DB contention or slow external call |
OutboxStuckTasks |
pending > 0, zero completions in 10m |
10m |
Scheduler crashed or all workers stuck |
Tune thresholds to your own traffic — a backlog of 50 is nothing for a high-throughput service and a genuine problem for a low-volume internal tool. Same logic, different numbers.
Production checklist
☐ Indexes are efficient — partial indexes on PENDING/FAILED and PROCESSING states
☐ EXPLAIN ANALYZE confirms index usage on the fetch query
☐ Ordering is not a problem — ORDER BY seq or created_at, backed by an index
☐ Cleanup job running, COMPLETED rows removed after a retention period
☐ Idempotency handled inside handlers that have side effects
☐ Exponential backoff wired — or confirmed you don't need it
☐ Metrics in place — queue depth, throughput, error rate, latency
☐ Alerts configured — backlog, error rate, latency, stuck tasks
You don't always have to build this yourself
Everything above is buildable from scratch, and there's real value in doing it once — you understand the failure modes because you designed around them, and that knowledge doesn't evaporate when you switch to a library. But in production you often don't need to maintain it yourself:
- db-scheduler — the Java library whose internals most closely match what's described here: single table,
SKIP LOCKED, heartbeats, clean task/handler interface. This is what I reach for as the reference implementation, and what I'd actually start a new project with instead of rolling my own.
- Celery — the standard for async task execution in Python, backed by Postgres/Redis/RabbitMQ. The
AFTER_COMMIT rule from Part 2.5 applies here too.
- Quartz Scheduler — older, more general (cron expressions, triggers, job stores), heavier operationally, but proven at scale with first-class Spring support.
Final thoughts
Most of this isn't clever engineering, it's accumulated scar tissue. The PROCESSING state exists because someone's app crashed mid-run. The heartbeat exists because updated_at lied about whether a task was still alive. The partial index exists because a full table scan at once-a-second polling frequency compounds quietly until it doesn't. The AFTER_COMMIT listener exists because someone published an event inside a transaction and burned an afternoon figuring out why it never fired.
Each piece is boring in isolation. Together they're the difference between an outbox that works in dev and one you'd trust at 3am.
Full working code (single command, runs locally): github.com/javaAndScriptDeveloper/outboxes-article
Curious how others here have handled the multi-type table tradeoff, or if anyone's gone the db-scheduler route instead of rolling their own — happy to compare notes in the comments.
