EDA operations: delivery semantics, idempotency, and DLQ 🚨

If EDA is the cool part, this is the part that saves you in production. Spoiler: it is more important.

Delivery semantics: what are you really promising? 📦

The classic labels:

• At-most-once: some events may be lost, but there will be no duplicates.
• At-least-once: events are not lost (ideally), but you may have duplicates.
• Exactly-once: the most ambitious goal. Kafka supports EOS (Exactly-Once Semantics) via idempotent producers and transactions, but this guarantee only applies between Kafka topics (Kafka-to-Kafka operations). As soon as you leave the cluster—write to a database, call an API, update another store—the EOS guarantee no longer applies. In cross-system scenarios, what you really get is effectively-once: application idempotency plus deduplication.

Practical rule: in distributed architectures, assume duplicates and design accordingly. The broker’s EOS does not absolve you from consumer idempotency.

Idempotency: making consumers “replay-safe” 🛡️

A consumer is idempotent if processing the same event multiple times yields the same result.

Common strategies:

• dedup store: save processed event_ids and discard duplicates;
• upsert on a business key (for example order_id) instead of a pure insert;
• controlled side effects: send emails with application-level exactly-once semantics via lock/dedup.

Ordering and partitioning: order is not free 🧵

Many platforms guarantee order only:

• within a partition;
• for events with the same key.

So the real question becomes: what is the right key?

• for orders: often order_id;
• for users: often user_id.

If you choose a random key, ordering becomes a poetic concept.

Error handling: retry, backoff, DLQ 🧯

    flowchart LR
  B[(Broker)] --> C[Consumer]
  C -->|OK| ACK[Ack / Commit]
  C -->|Fail| R{Retry
< max?}
  R -->|Yes| B
  R -->|No| DLQ[(DLQ)]
  DLQ --> OPS[Runbook
triage + reprocess]
  

A robust model includes:

• retry with backoff (to avoid storms);
• limit attempts;
• DLQ (Dead Letter Queue) for “poison” messages;
• alerting on DLQ and consumer lag.

Classifying errors and DLQ: not a trash can, a process 🧠

Not all errors deserve the same treatment. Transient errors (network, timeout, slow dependencies) require retry with backoff; permanent errors (invalid schema, corrupted data) require stop, send to DLQ, and triage. Deciding this during an incident is the best way to make the incident worse.

Example: if a consumer fails due to a timeout talking to an external service, try retry with backoff and circuit breaker; if it fails due to an unrecognized schema, send it to DLQ and alert for manual triage.

The DLQ is not a silent limbo: it is useful only if there is an active process supervising it. Minimum conditions:

• you can distinguish transient vs permanent errors (and decide this beforehand, not during an incident);
• you have a runbook: who watches, with what priority, how to reprocess;
• you can fix and replay safely (idempotency—always, not “probably”).

Minimal runbook for DLQ:

• immediate alert when a message lands in DLQ;
• automatic classification (schema error, transient, business error) where possible;
• assignment to an owner and a procedure for reprocessing with verified idempotency;
• DLQ metrics in the main dashboard, not in a hidden tab.

Consumer lag and backpressure 📈

In EDA, often you do not have errors—you have delays.

Consumer lag measures how many events the consumer has not yet processed compared with the latest offset published. Rising lag is the first signal that the consumer cannot keep up: a slow dependency, CPU saturation, performance bug, or simply increased traffic. In Kafka, lag is measured per partition and consumer group; in other brokers the concept is similar.

Backpressure is the pressure that builds upstream when a consumer slows down. If unmanaged, it propagates to the producer (which keeps writing) and can degrade the whole system in cascade. Some platforms offer native backpressure mechanisms; otherwise you need to manage it explicitly.

Practical strategies for coping with lag peaks:

• horizontal scaling of the consumer group (more instances = more parallelism, within the partition limit);
• batch processing to amortize per-event overhead for expensive operations;
• circuit breaker toward slow dependencies, to avoid an external service draining all capacity silently;
• alerting on lag thresholds (for example “lag > X for more than Y minutes”) with graduated escalation.

Typical metrics to monitor:

• consumer lag (per topic/partition and consumer group);
• throughput (input vs output, to expose the delta);
• error rate by type (transient vs permanent);
• average and p99 processing time per event.

End-to-end observability: correlation ID or chaos 🧬

Without correlation, debugging an asynchronous flow is like searching for a needle in a haystack… in the dark… while the haystack burns.

Minimum practices:

• correlation_id propagated through events and logs;
• distributed tracing when possible;
• dashboards for business flows (not just technical metrics).

If you want a standard base, OpenTelemetry helps correlate signals, but application discipline is still required.

Responsibilities of producer and consumer 🤝

A serious producer publishes coherent events with mandatory metadata and stable semantics. A serious consumer assumes duplicates, handles out-of-order events when necessary, and fails observably. When this responsibility is not explicit, the system “works” until it scales.

What to expect from a responsible producer:

• publishes events with a stable schema and explicit versioning;
• always includes event_id, timestamp, and correlation_id in metadata;
• does not break schema compatibility without notice and without versioning;
• documents the event semantics (what it means, when it is emitted, and what it does not mean).

What to expect from a responsible consumer:

• does not assume message uniqueness: implements deduplication or idempotency;
• does not assume global order: handles out-of-order events at least for known scenarios;
• logs every error in a structured way and ties it to correlation data;
• exposes health metrics (lag, error rate, throughput) without anyone having to hunt for them.

When these contracts are not written anywhere—in ADRs, runbooks, or topic docs—they are “negotiated” during the first production incident. That is a suboptimal time.

Minimal operational checklist ✅

There is no “best practice from slides” here: these are the minimum building blocks for avoiding turning retries into an incident printer. If you have to choose what to do first, do these first.

• Retry policy documented for every consumer.
• DLQ with ownership and runbook.
• Verified idempotency (tests and design review).
• Dashboard: lag, errors, DLQ, throughput.
• Structured logging + correlation ID.

Next steps 🚀

Once operations are in place, you can start designing multi-step flows and migrations without living in fear of dual-write.

• For Outbox/Saga and incremental migrations: see the production patterns guide.
• For governance and event catalog: see the governance guide.