Messaging 101: queue vs stream vs pub/sub 📬

If your team has the phrase “let’s put Kafka” as a universal answer, this page is here to reduce the damage.

The point is not the tool: it is the operational model 🧠

When you choose a messaging system, you are choosing:

• how to handle durability and replay;
• what guarantees you have on ordering;
• how you scale consumers;
• how you handle errors and retry.

Queue 🧺

    flowchart LR
  P[Producer] --> Q[(Queue)]
  Q --> C[Consumer]
  C -->|ack| Q
  

Model: messages are consumed and conceptually removed after the ack.

• Great for: asynchronous tasks, distributed jobs, competing consumer workloads.
• Watch out for: DLQ, visibility timeout, retry storms, poison messages.

Stream 🧾

    flowchart LR
  P[Producer] --> S[(Stream / Log)]
  S --> CG1[Consumer group A]
  S --> CG2[Consumer group B]
  CG1 -->|offset| S
  CG2 -->|offset| S
  

Model: append-only log with retention; consumers keep an offset.

• Great for: multiple integrations, audit/replay, data pipelines, projections (CQRS).
• Watch out for: partitioning, reprocessing, schema evolution, consumer lag.

Pub/sub 📣

    flowchart LR
  P[Publisher] --> T[(Topic)]
  T --> S1[Subscriber 1]
  T --> S2[Subscriber 2]
  T --> S3[Subscriber 3]
  

Model: publish and fan-out to subscribers (often with filters).

• Great for: notifications, integrations to many consumers, near real-time triggers.
• Watch out for: filters, rate limits, backpressure, topic governance.

Things that always confuse people (and should not) 🧩

Ordering 🧵

When someone says “we need ordering,” the real question is: ordering of what, where, and with which key? Many brokers guarantee it, but under very specific conditions.

• it is often guaranteed only per key/partition;
• if you change the partitioning key mid-project, prepare to explain why order is no longer a constitutional right.

Consumer group 🧑‍🤝‍🧑

It is the basic tool for scaling: more consumers, more throughput. But it brings concurrency, race conditions, and “why did the event arrive later?” (spoiler: it did not arrive later; you processed it later).

• Same group: consumers share the work—each message is delivered to one consumer only (competing consumers). This is the horizontal scaling model for distributing load.
• Different groups on the same topic: each group receives all the messages independently (fan-out). Consumer groups A and B both read the full stream without interfering with each other.

Summary:

• scales horizontally by dividing the work;
• requires careful key design and acceptance that more consumers means more concurrency and more edge cases.

Retention and replay 🔁

Replay is power (rebuild state, replay projections, fix bugs), but it is also a maturity test: if you are not idempotent, it becomes roulette.

• if you can replay, you must design idempotency;
• if you cannot replay, you must invest even more in reliability and error handling.

Practical criteria for choosing 🎛️

Ask yourself:

1. Do you need replay to rebuild state or projections? → stream.
2. Do you need to distribute jobs and do not care about the full history? → queue.
3. Do you need clean fan-out to many consumers? → pub/sub (or stream with multiple consumer groups).
4. Do you have strong ordering constraints? → verify how it works in practice in the chosen broker.

Three capabilities that are often underestimated 🛠️

In the “broker A vs broker B” debate, people often look only at throughput and price. Then the real problems appear and you discover that three questions were missing:

• Filtering: can the broker filter events at the infrastructure level, or do you have to download everything and filter in the consumer? If it supports server-side filters (for example by header or attribute), you save bandwidth and CPU; otherwise implement filters in the consumer and monitor drops.
• Backpressure/flow control: how do you slow producers or consumers when the system saturates? Prefer mechanisms that push pressure back to the producer (rate limiting, pause) instead of uncontrolled retries; sometimes a temporary buffer with a shed policy is more practical.
• Guaranteed delivery (practical, not marketing): what guarantees do you really have on persistence, ack, retries, and duplicates? Test practical cases (network partitions, broker crashes) and define what happens at the application level; do not accept marketing claims without recovery tests.

These three things are not “nice to have”: they often determine operational cost, latency, and on-call peace of mind.

Publish/subscribe, point-to-point, and request/reply 📮

These labels describe how messages travel—regardless of which broker you use.

• Publish/subscribe: an event is delivered to all interested subscribers (fan-out). This corresponds to the model of multiple consumer groups on the same topic.
• Point-to-point: a message goes to a single recipient (competing consumer). This corresponds to multiple consumers in the same group on a queue or partition.
• Request/reply: a producer waits for a response from a specific recipient. In EDA it is useful when deterministic response is needed, but be careful: if you are only moving synchrony under a carpet of queues, you will eventually trip over it.

Practical rule: if you are recreating request/response with a queue, you are probably looking for a direct HTTP/gRPC call.

Minimal adoption checklist ✅

This is not bureaucracy: it is insurance. Without these explicit decisions, the platform falls into “everyone does what they want” within about two sprints.

• Define naming for topics/streams.
• Define ownership (who is responsible for the channel).
• Define policy: retention, DLQ, retry.
• Define metrics: throughput, error rate, consumer lag.

Next steps 🚀

If you understand how messages move, the next step is to design what they contain and how to make them operational.

• To design payloads and schemas: see the event design guide.
• For idempotency, retries, and DLQ: see the operational guide.