Two-Layer Idempotency: Why Not Every Settlement System Needs an Outbox

A few weeks ago I published a write-up on inverting the control flow of a settlement API: instead of the endpoint owning window semantics (batching, cutoffs, retries), it became a pure write-and-forget operation, and all of that coordination logic moved to a cold-path scheduler. One of the best responses I got was from a fellow engineer who pushed on the exact point that matters most in any event-driven settlement system: what happens to the message between the moment your transaction commits and the moment the broker acknowledges it?

That question is worth a full article on its own, because the answer we landed on — a two-layer idempotency model instead of a transactional outbox is a trade-off I think gets under-discussed. Most write-ups on reliable event publishing jump straight to "use an outbox," treat it as a solved problem, and move on. It's a great pattern. It's also not free, and for a meaningful class of systems, it's more machinery than the failure mode actually justifies.

The gap everyone is trying to close

The problem is well known: you commit a database transaction, and then you need to tell the world about it via Kafka. If you publish inside the transaction, you risk publishing an event for work that later rolls back. If you publish after the transaction commits, in something like Spring's afterCommit(), you close that hole, but you open a new one: the process can crash, or the network can fail, in the narrow window between the commit finishing and the broker acknowledging the message. Now your database says the work happened, but no one downstream knows about it.

The outbox pattern solves this cleanly. You write the event to an outbox table in the same transaction as your business data, so the write is atomic with the state change. A separate poller or CDC process then reads the outbox and publishes to Kafka, retrying until it succeeds, and only then marks the row as sent. It gives you a real at-least-once guarantee without needing two-phase commit across a database and a broker.

I used exactly this pattern on another project, an Open Finance Brasil report service, where audit correctness on consent events was non-negotiable and the extra moving parts were worth it. So this isn't a case of not knowing the pattern or not trusting it. It's a case of asking whether it's the right tool for this system.

Why I didn't reach for it here

For a settlement system built around STR, the operative constraint is the size of the settlement window, which is measured in minutes, not milliseconds. A few seconds of relay latency from an outbox poller is genuinely negligible against that budget, my correspondent was right about that, and it's the strongest argument in the outbox's favor.

But the outbox isn't just a table. It's a second scheduler, a poller or CDC pipeline, and a relay process that now needs its own monitoring, its own failure handling, and its own on-call story. That's a permanent addition to the system's operational surface, taken on to close a failure window that, for this domain, is both rare and cheap to detect after the fact.

So the trade-off I made was deliberate: accept the dual-write gap between afterCommit() and broker acknowledgment, and instead of preventing it at the producer, absorb it at the consumer.

Layer one: afterCommit() eliminates phantom messages

The first layer is about correctness of causality, not delivery. Publishing inside afterCommit() guarantees that a message is never sent for a transaction that didn't actually happen. If the business transaction rolls back, nothing is ever published — no compensating logic, no "unpublish," no phantom event downstream needs to reconcile. This layer doesn't touch the crash-between-commit-and-ack scenario at all; it exists purely to rule out the opposite failure, publishing something that turns out to be false.

@Transactional(propagation = Propagation.REQUIRES_NEW)
public void process(SettlementWindow window, LocalDate today, Participant participant) {

    // ... validation and batch assembly omitted ...

    FileBatch savedBatch = batchPort.save(batch);
    orderPort.updateStatusBatch(ordersWithBatch);

    if (TransactionSynchronizationManager.isActualTransactionActive()) {
        TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
            @Override
            public void afterCommit() {
                publisherPort.publish(savedBatch);
            }
        });
    } else {
        publisherPort.publish(savedBatch);
    }
}

Layer two: SHA-256 + Redis catches duplicate delivery

The second layer is where the actual reliability work happens, and it lives entirely on the consumer side. Spring Kafka's producer will retry a send if it doesn't get an acknowledgment, which is exactly the right behavior for the crash scenario my correspondent raised: if the process dies after afterCommit() but before the ack comes back, the retry may result in the broker actually having received the message once already, with only the acknowledgment having been lost. The consumer, not the producer, is what needs to be defensive here.

Before processing a batch, the consumer computes a SHA-256 checksum of the incoming message and checks it against Redis. If the key already exists, the message is a duplicate produced by a retry, and it's discarded. If it doesn't, the consumer writes the key and proceeds. This is a small, fast, stateless check that turns "at-least-once delivery" into "effectively-once processing" without needing exactly-once semantics from the broker at all.

public void onMessage(ConsumerRecord<String, byte[]> record) {
    String checksum = sha256(record.value());
    String key = "dedup:" + checksum;

    Boolean isNew = redisTemplate.opsForValue()
            .setIfAbsent(key, "1", Duration.ofHours(24));

    if (Boolean.FALSE.equals(isNew)) {
        log.warn("Duplicate message detected, checksum [{}]. Skipping.", checksum);
        return;
    }

    processBatch(record.value());
}

Why the two layers are complementary, not redundant

Neither layer solves the whole problem alone. afterCommit() alone still leaves you exposed to duplicate delivery from producer retries. A dedup filter alone, without afterCommit(), would happily deduplicate a phantom message that should never have existed. Together, they cover the two failure modes that actually matter for this system: never publish something false, and never process something twice.

What this buys is a system that's honest about where it accepts risk. It isn't as airtight as a full outbox with a transactional relay — I'll say that plainly, because it's true, and pretending otherwise would be dishonest about the trade-off. But airtight isn't the goal; matching the mechanism to the failure cost is. If an event is somehow lost entirely (a scenario the two layers above don't even fully rule out, since they handle duplication and phantom writes, not total loss), the recovery path is a simple query against a BATCHED status field. That's a five-minute manual fix, not an incident.

The actual lesson

The interesting part of this exchange wasn't the specific pattern, it was the reasoning behind rejecting a "better" one. An outbox is strictly stronger than a producer-side commit hook plus a consumer-side dedup filter. But strictly stronger doesn't mean strictly better for a given system, because every additional guarantee has an operational cost, and that cost has to be weighed against what a lost or duplicated event actually costs you.

For a settlement window measured in minutes, with a trivial recovery query, the outbox's extra guarantee wasn't buying enough to justify a second scheduler and relay process living in production. For the Open Finance report service I mentioned earlier, where audit correctness on every single consent event is the entire point of the system, that same outbox was worth every bit of its complexity.

Idempotency isn't a single pattern you either apply or don't. It's a budget, and the right layering depends entirely on what you're protecting and how much a mistake actually costs.