Skip to content
V3.0 // STABLE
LOAD 12%
LAT 24MS
SLA 99.99%
wahyu amirulloh

The Transactional Outbox Pattern: Distributed Consistency

•3 min read•
4 views
transactional outboxconsistenykafkadistributed systems

In a microservices architecture, we often need to update a database and publish an event to a message broker (like Kafka) simultaneously. However, distributed transactions (2PC) are expensive and often impossible. This is where the Transactional Outbox Pattern shines.

The Problem: Dual Write Challenge

What happens if your service successfully updates the database but crashes before it can publish the event to Kafka? Your system is now Inconsistent.

[!CAUTION] Ghost Events: If you publish to Kafka before the DB commit and the DB commit fails, you've sent an event about a change that didn't happen.

The Solution: One Atomicity

Instead of trying to talk to two separate systems (DB and Kafka), we only talk to the DB. We store the event in a dedicated outbox table in the same database transaction as our business logic.

Live architecture
Analyzing Schema...

Arch Note

Interactive logic enabled. Click components in expanded view for technical service definitions.

Layer.0 / Distributed_System_Viz

Step-by-Step Implementation

  1. Atomicity: Your service starts a database transaction.
  2. Double Write: It updates the "Orders" table AND inserts the event into the "Outbox" table.
  3. Commit: The transaction is committed.
  4. Relay Process: A separate, asynchronous "Outbox Relay" process polls the Outbox table for new entries and publishes them to Kafka.
  5. Acknowledge: Once published to Kafka, the entry in the Outbox table is marked as processed or deleted.

Consistency Comparison

ApproachConsistencyComplexityResult
Async (Post-Commit)Best EffortLowLikely data loss on crash.
Sync (Pre-Commit)InconsistentMediumPhantom events on DB rollback.
Transactional OutboxStrict/AtomicHighReliable exactly-once/at-least-once.

Consultant's Choice: Polling vs. CDC

  • Polling: Simple. Run a SELECT query every second. Good for low/medium volume.
  • Change Data Capture (CDC): Advanced. Use tools like Debezium to listen to the DB binlog/WAL and stream them directly to Kafka. This has near-zero overhead on the DB.

[!TIP] Pro-Tip: Ensure your event consumers are Idempotent because the Outbox Relay might occasionally publish the same event twice due to a network timeout during the acknowledgment phase.

Mastering the Outbox pattern is the difference between a "working" microservices mesh and a "resilient" microservices mesh.