Production patterns: Outbox, Saga, and incremental migration 🧰

This guide is for the moment when you move from “how nice, events!” to “okay, but how do I do this without losing data?”

Transactional Outbox: the dual-write problem 🧩

    flowchart LR
  S[Service] -->|1. tx| DB[(Database)]
  DB -->|2. insert| O[(Outbox table)]
  PUB[Outbox publisher] -->|poll| O
  PUB -->|publish| B[(Broker / Stream)]
  

Classic problem:

1. write to the database (transaction OK)
2. publish the event to the broker

If you die between (1) and (2) (crash, network, timeout), the database and broker drift out of sync.

Solution: Outbox 📦

The idea is simple and slightly boring (which is why it works): instead of trying to publish events after the transaction and hoping not to die between one step and the next, make the event part of the commit. Then publish reliably with retries without inventing distributed magic.

• in the same database transaction, also write a record to an outbox table;
• a separate publisher reads the outbox and publishes to the broker;
• after publish, mark the record as sent.

Result: no ghost events and no updates without an event.

Saga: distributed transactions without ACID illusions 🎢

    flowchart LR
  E1[OrderCreated] --> I[Reserve inventory]
  I --> OKI{OK?}
  OKI -->|Yes| P[Authorize payment]
  OKI -->|No| C1[Compensate: cancel order]
  P --> OKP{OK?}
  OKP -->|Yes| CONF[Confirm order]
  OKP -->|No| C2[Compensate: release inventory]
  

When a process spans multiple services, avoid the temptation to use two-phase commit everywhere. Two-Phase Commit introduces a coordinator as a single point of failure and a blocking protocol: in modern distributed systems it tends to reduce availability and increase latency far more than the benefits justify.

A Saga models the process as:

• a sequence of steps;
• each step publishes an event or receives a command;
• if something fails, compensations trigger (business rollback).

Conceptual example:

• OrderCreated → reserve inventory
• InventoryReserved → authorize payment
• if payment fails → ReleaseInventory

The point is not “to roll back perfectly,” but to preserve business integrity and eventual consistency.

Choreography-based Saga vs Orchestration-based Saga 🎭

Sagas are implemented in two main variants—the same distinction described in the pattern guide:

• Choreography-based Saga: each service reacts to events and publishes new events for the next step. There is no central control point. Pro: decentralized, lower coupling. Con: the business flow is spread across services and can be hard to trace without observability tools.
• Orchestration-based Saga: an orchestrator (often a workflow engine or dedicated service) explicitly drives each step—emits commands, waits for results, manages timeouts and compensations. Pro: process visibility, centralized error handling. Con: the orchestrator becomes a critical component and introduces more coupling.

The choice is not obvious: use choreography for simple processes with independent side effects; use orchestration when you need explicit control over timeouts, retries, and coherent compensations.

Incremental migration 🌱

Migrate to EDA with a big bang is an excellent way to collect tragic anecdotes.

An effective pattern for gradual migration is the Strangler Fig (Martin Fowler): introduce an intermediate layer (proxy or facade) that intercepts requests and routes traffic between the legacy system and the new event-driven system. As features are migrated, the routing gradually shifts toward the new system until the legacy can be turned off without a big bang.

    flowchart LR
  CLIENT[Client] --> PROXY[Proxy / Facade]
  PROXY -->|migrated features| NEW[New EDA system]
  PROXY -->|legacy features| OLD[Legacy system]
  NEW -->|events| BROKER[(Broker)]
  

Typical incremental approach:

1. publish events from the existing system (without changing the core path);
2. add new consumers that perform side effects in parallel;
3. observe, stabilize, add governance;
4. migrate features piece by piece through the proxy layer, turning off legacy integrations.

Publish events even before consumers exist (at the beginning) 🚦

Yes, it can make sense to publish events before there are real consumers: it allows you to stabilize naming, schema, and metadata, and to understand the real traffic volume. The important thing is not to cheat: even in this phase you need ownership, clear retention, and minimal monitoring. Otherwise you are just accumulating technical debt with enthusiasm.

Practical checks to put in place immediately:

• monitor volume and alert on unexpected spikes;
• minimum retention and clear policy to avoid useless buildup;
• ownership defined from the start (even if consumers do not exist yet).

Practical test before enabling replay: publish events in staging, verify that test consumers are idempotent, and ensure replay does not generate unexpected load on downstream systems.

Rollback strategy 🪂

It seems boring, so it is often skipped. Then it becomes suddenly interesting.

• Keep both the legacy flow and the event-driven flow for a while.
• Define when a consumer can be deactivated without losing integrity.
• Verify idempotency before enabling replay.

“Production-ready” checklist ✅

This checklist will not make you “enterprise-ready,” but it will help you avoid the classic mistakes when EDA meets real data and real failures.

Outbox and dual-write

• Do you have a plan for dual-write? (Outbox or equivalent—not “write to the DB and then to the broker, hoping it works”)
• Does the Outbox publisher handle failures with retry and backoff?
• Are Outbox records marked as sent and then removed or archived with an explicit retention policy?

Saga and compensation

• Do you have a model for multi-step failures? (Saga + documented compensations)
• Are compensations idempotent? (can they be invoked multiple times without harm)
• Have you explicitly chosen between choreography and orchestration—and do you know why?
• Is there a timeout on every Saga step? Are stuck steps detected and handled?

Incremental migration

• Are you using a proxy/facade layer (Strangler Fig) to isolate the legacy?
• Do you have separate metrics for legacy and EDA traffic so you can understand when it is safe to turn one off?
• Is rollback documented: do you know how to return to the legacy flow if something goes wrong?
• If you publish events before consumers exist, do you already have retention, ownership, and monitoring in place?

Observability and operations

• Do you have metrics and alerts for lag and DLQ?
• Does every event have event_id, correlation_id, and timestamp in the metadata?
• Have you planned replay—and verified that consumers are idempotent before enabling it?
• Have you tested replay in staging and measured the load it generates on downstream systems?

Next steps 🚀

If you are approaching EDA incrementally, the next bottleneck almost always becomes the contract: naming, schema evolution, and governance.

• For contracts and schema evolution: see the event design guide.
• For governance and the event catalog: see the governance guide.