Outbox Relay Retries: At-Least-Once Delivery Without Duplicate Events

A practical Spring Boot and MySQL design for retrying transactional Outbox events safely, handling at-least-once delivery, duplicate consumers, leases, backoff, and dead-letter recovery.

Transactional Outbox solves one difficult part of a distributed workflow: it lets a service commit business data and the event describing that change in the same database transaction. It does not, by itself, guarantee that a relay will deliver the event exactly once.

The failure that matters is small but common. The relay publishes an event, loses its connection before marking the row as sent, and retries the same row after it restarts. The consumer sees the event twice. A reliable design accepts that delivery is usually at least once and makes the consumer safe to run more than once.

This article builds a small MySQL and Spring Boot design around that failure. It covers the Outbox table, a claim-and-publish loop, exponential backoff, a dead-letter state, and an idempotent consumer. The examples are intentionally compact so they can be adapted to an existing service.

What the Outbox does and does not guarantee

Suppose an order service writes an order and an OrderCreated event. Without an Outbox, the database commit and the broker publish are two independent operations. A process crash between them can leave an order in the database with no event, or an event with no matching order.

The Outbox puts both writes in one local transaction:

START TRANSACTION;

INSERT INTO orders(id, customer_id, total_amount, status)
VALUES ('ord-1007', 'cus-18', 129.00, 'PAID');

INSERT INTO outbox_event
    (event_id, aggregate_type, aggregate_id, event_type, payload,
     status, attempts, available_at, created_at)
VALUES
    ('evt-7c2f', 'ORDER', 'ord-1007', 'OrderCreated',
     '{"orderId":"ord-1007","total":129.00}',
     'PENDING', 0, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP);

COMMIT;

If the transaction rolls back, neither row is visible. If it commits, the event remains durable until a relay delivers it. The missing guarantee is the boundary after the broker accepts the event and before the relay records a successful delivery. There is no portable transaction that covers both a relational database and every message broker.

A state machine that makes retries visible

Use explicit states instead of a single boolean flag. A useful starting point is:

  • PENDING: ready for a relay to claim;
  • PROCESSING: claimed by a worker and protected by a lease;
  • SENT: the broker publish was acknowledged;
  • DEAD: the retry budget was exhausted and an operator must inspect it.

The lease is important. A worker can disappear while processing a row, so another worker must be able to reclaim it after a timeout.

CREATE TABLE outbox_event (
    event_id       VARCHAR(64)  PRIMARY KEY,
    aggregate_type VARCHAR(64)  NOT NULL,
    aggregate_id   VARCHAR(64)  NOT NULL,
    event_type     VARCHAR(128) NOT NULL,
    payload        JSON         NOT NULL,
    status         VARCHAR(16)  NOT NULL,
    attempts       INT          NOT NULL DEFAULT 0,
    available_at   DATETIME     NOT NULL,
    locked_until   DATETIME     NULL,
    last_error     VARCHAR(1000) NULL,
    created_at     DATETIME     NOT NULL,
    sent_at        DATETIME     NULL,
    INDEX ix_outbox_ready (status, available_at),
    INDEX ix_outbox_lease (status, locked_until)
);

Keep event_id stable across every retry. It is the identity that the consumer uses for deduplication; generating a new ID for each attempt defeats that purpose.

Claim rows without two workers doing the same work

A relay should claim a small batch, not scan the entire table on every tick. MySQL 8 supports FOR UPDATE SKIP LOCKED, which lets concurrent workers skip rows already locked by another worker.

START TRANSACTION;

SELECT event_id
FROM outbox_event
WHERE (status = 'PENDING' AND available_at <= CURRENT_TIMESTAMP)
   OR (status = 'PROCESSING' AND locked_until < CURRENT_TIMESTAMP)
ORDER BY created_at
LIMIT 50
FOR UPDATE SKIP LOCKED;

UPDATE outbox_event
SET status = 'PROCESSING',
    locked_until = DATE_ADD(CURRENT_TIMESTAMP, INTERVAL 60 SECOND),
    attempts = attempts + 1
WHERE event_id IN (...);

COMMIT;

The claim transaction is deliberately separate from the broker call. Holding a database lock while waiting on a network request creates long transactions and reduces throughput. The lease protects the row while the worker is away from the database.

A small Spring Boot relay

The relay publishes the stored payload and then marks the event as sent. The important detail is that the mark happens only after the broker client reports success.

@Scheduled(fixedDelayString = "${outbox.poll-ms:1000}")
public void relayBatch() {
    List<OutboxEvent> events = repository.claimBatch(50, Duration.ofSeconds(60));

    for (OutboxEvent event : events) {
        try {
            publisher.publish(
                event.eventType(),
                event.eventId(),
                event.payload()
            ).get(5, TimeUnit.SECONDS);

            repository.markSent(event.eventId(), Instant.now());
        } catch (Exception ex) {
            repository.scheduleRetry(
                event.eventId(),
                nextAttemptAt(event.attempts()),
                abbreviate(ex.getMessage())
            );
        }
    }
}

private Instant nextAttemptAt(int attempts) {
    long seconds = Math.min(900, 1L << Math.min(attempts, 10));
    return Instant.now().plusSeconds(seconds);
}

Production code should also add a small random jitter to the delay. Without jitter, a large group of rows that failed at the same time can all wake up together and create another load spike.

When does an event become dead?

Retries are useful for transient failures: a broker timeout, a short network interruption, or a temporarily unavailable downstream service. They are not useful for a permanently invalid payload. After a bounded number of attempts, move the row to DEAD and preserve the last error.

UPDATE outbox_event
SET status = CASE
                 WHEN attempts >= 12 THEN 'DEAD'
                 ELSE 'PENDING'
             END,
    available_at = :nextAvailableAt,
    last_error = :lastError,
    locked_until = NULL
WHERE event_id = :eventId
  AND status = 'PROCESSING';

A dead event needs an operational path. Expose a small admin report, alert on the count and age of DEAD rows, and provide a deliberate replay action. Replaying should return the same event_id to the consumer so it remains idempotent.

The consumer must tolerate duplicates

Even a careful relay can publish twice. For example, the broker accepts the message and the relay process crashes before markSent commits. The next worker publishes the same event again. The consumer should record the event ID in the same transaction as its business change.

CREATE TABLE processed_event (
    consumer_name VARCHAR(80) NOT NULL,
    event_id      VARCHAR(64) NOT NULL,
    processed_at  DATETIME NOT NULL,
    PRIMARY KEY (consumer_name, event_id)
);
START TRANSACTION;

INSERT INTO processed_event(consumer_name, event_id, processed_at)
VALUES ('inventory-service', 'evt-7c2f', CURRENT_TIMESTAMP);

-- A duplicate raises a duplicate-key error and skips the business update.
UPDATE inventory
SET reserved = reserved + 1
WHERE sku = 'sku-42';

COMMIT;

The insert and the business update must share a transaction. If the consumer records the event first and crashes before applying the update, a later delivery would be ignored even though the business change never happened.

Failure tests worth running

A happy-path test only proves that the relay works when nothing goes wrong. Add failure injection around the two boundaries that matter:

  1. Crash before the publish call. The event should be reclaimed after the lease expires.
  2. Return success from the broker, then crash before markSent. The event will be published twice, but the consumer should apply it once.
  3. Return a timeout for the first three attempts. The row should move through scheduled retry times and eventually return to PENDING.
  4. Return a permanent validation error. The row should become DEAD after the configured limit.
  5. Replay a dead event. The same event ID should make the consumer a no-op when it was already processed.

For each test, capture the Outbox row, broker delivery count, consumer table, and final business state. That evidence is more useful than a statement that the system is “exactly once.” In this design, the honest guarantee is at-least-once delivery with idempotent effects.

Metrics that find a stuck relay

Track the age of the oldest pending event, the number of processing leases that expired, publish latency, retry count, dead-event count, and duplicate-consumer count. A relay can be running and still be unhealthy if the oldest event keeps getting older.

Useful alerts include:

  • oldest PENDING event older than the normal processing window;
  • the DEAD count increases;
  • broker publish failures exceed a short threshold;
  • duplicate events rise sharply after a deployment.

How this fits with the rest of the system

The Outbox relay is one part of a larger workflow. For a multi-step order process, the transaction and compensation trade-offs are covered in Distributed Transactions in Practice. If the worker protects a shared inventory operation, compare the lease approach with Java distributed locks and fencing tokens. For SQL queries that read the resulting order data, see JOIN vs EXISTS in SQL.

Checklist

  • Write the business row and Outbox row in one local transaction.
  • Keep one stable event ID for every retry and replay.
  • Claim short batches with a lease and avoid holding locks during network calls.
  • Use bounded exponential backoff with jitter.
  • Move permanently failing rows to a visible DEAD state.
  • Record the event ID with the consumer business update in one transaction.
  • Test the crash after publish and before marking the row as sent.
  • Monitor event age, retry volume, dead rows, and duplicate deliveries.

The practical goal is not to pretend that a database and a broker share one atomic commit. It is to make every failure observable, retryable, and safe for the business operation to encounter more than once.

Leave a Reply

Your email address will not be published. Required fields are marked *