Understand log-based event streaming: topics, partitions, offsets, consumer groups, retention, compaction and replay, and when to choose a log over a message queue.
What you'll learn: By the end of this lesson you will be able to explain how an append-only log differs from a queue, describe topics, partitions, offsets and consumer groups, state the ordering and delivery guarantees a partitioned log really gives, and decide between a queue and a stream for a given workload.
A queue is like a to-do tray: when you finish a task, it leaves the tray. A log is more like a ledger or a security-camera tape. Entries are written at the end in order, they stay for a set time, and any number of people can read the tape, each at their own position, and rewind it whenever they like. Reading does not erase anything.
That one change, keeping events instead of deleting them on delivery, opens up many independent consumers of the same data, replay of history to rebuild a system, and very high throughput. It also changes how you think about ordering, scaling and failure. This lesson builds the vocabulary used by Apache Kafka and similar systems, though the ideas apply to any log-based platform. It assumes you have read message queues.
The core idea
An event stream is a durable, ordered, append-only log that consumers read by position. Producers append events to the end. The system keeps them for a configured retention. Each consumer remembers its own offset, the number of the next event it will read, so every consumer progresses independently and can move its offset to re-read history.
In a queue the broker tracks per-message delivery state and deletes messages once acknowledged. In a log the broker mostly stores and serves a sequence, and each consumer owns its position (the broker often stores the committed offset on its behalf). That keeps the broker simple and fast, and makes replay almost free.
The building blocks
An event is an immutable record that something happened (OrderPlaced, PaymentFailed), with a key, a value and a timestamp. A topic is a named stream of one kind of event, such as orders or page-views.
One log on one machine would limit throughput and storage, so a topic is split into partitions, each an independent ordered log spread across brokers. A producer picks the partition, usually by hashing the event's key, so all events with the same key land in the same partition. Events without a key spread across partitions and have no per-key ordering. Each event in a partition has a sequential offset (0, 1, 2, ...), and (partition, offset) identifies an event; offsets mean nothing across partitions.
Each partition is normally replicated to several brokers: one leader handles reads and writes, followers copy it, and if the leader's broker fails a follower is promoted. How many replicas must confirm a write before it counts as committed trades latency against durability, in the spirit of replication.
Each partition is its own ordered log with its own offsets; the key hash decides which one an event joins.
Consumer groups: scaling reads
A consumer group is a set of consumer instances cooperating to read a topic. Each partition is assigned to exactly one consumer in the group at a time, so members share the work and each event is processed by one member. Consequences:
Partitions cap parallelism within a group. With 6 partitions, 6 consumers work in parallel; a 7th sits idle.
Groups are independent. The fraud group and the analytics group each read every event at their own pace with their own offsets. This is publish/subscribe on top of stored data.
Rebalancing reassigns partitions when consumers join or leave. Processing may pause briefly, and a partition's new owner resumes from the last committed offset.
Committing offsets and delivery semantics
A consumer periodically commits its offset ("I have processed up to here"). If it crashes, a replacement starts from the last committed offset. When you commit relative to processing decides the guarantee: commit before processing gives at-most-once (a crash skips events); process then commit gives at-least-once (a crash re-reads and reprocesses), the usual default. For exactly-once effect, as with queues, no general guarantee exists across independent systems. Within some streaming platforms a transactional mode lets a pipeline atomically consume, update state and produce output events, so read-process-write cycles take effect once inside that platform. For effects outside it (an email, a payment API) you still need idempotent processing with a deduplication or idempotency key.
sequenceDiagram
participant B as Broker
participant C as Consumer
participant S as Sink
C->>B: poll from committed offset 41
B-->>C: events 41 to 50
C->>S: apply effects idempotently
S-->>C: done
C->>B: commit offset 51
Note over C,B: crash before commit means 41 to 50 are re-read
The commit comes last, so a crash replays the batch, which is why the sink must tolerate duplicates.
Ordering guarantees
Within a partition, order is total and preserved.
Across partitions, there is no ordering guarantee.
So choose the key so events needing relative order share a key. Use the order id as key and all events for one order are read in sequence, while unrelated orders run in parallel.
WARNING
Changing the number of partitions on an existing topic changes how keys map to partitions. Events for a key before and after the change may sit in different partitions, breaking per-key ordering for that transition. Provision headroom up front and treat repartitioning as a migration.
A producer can also reorder its own events if it retries a failed send while later sends succeed, unless configured otherwise (for example idempotent producer settings that deduplicate retries). Hot partitions are the cost of key routing: if one key (a bot, a default tenant) produces a large share of events, its partition becomes a bottleneck. Remedies are finer-grained keys, a salt for keys that do not need strict ordering, or handling the heavy key separately.
Retention, compaction and replay
A queue deletes a message after consumption; a log deletes by policy, independent of consumers.
Time or size retention. Keep events for, say, seven days, then drop the oldest segments. A consumer down longer than retention loses unread data, so retention must exceed the worst plausible consumer outage.
Log compaction. Keep at least the latest event for each key and eventually discard older ones (compaction runs in the background, so old events can linger). A compacted topic behaves like a table of current state and lets a new consumer rebuild a snapshot from the start. A tombstone (a key with an empty value) marks a deletion.
Indefinite retention. Cheap storage lets some teams make the log the system of record, which raises questions about privacy deletion and schema evolution.
Replay is the standout feature: rebuild a read model or search index from offset 0, fix a consumer bug and reprocess a time window, add a brand-new consumer that processes full history and then keeps up, or test a new service version against real past traffic. It requires idempotent consumers or a fresh target, and event formats understandable years later: use an explicit schema, add optional fields, avoid renaming, version events. The CQRS and event sourcing lesson goes deeper.
Streaming versus queue: how to choose
Dimension
Message queue
Event stream (log)
Core model
Broker tracks each message; deleted after ack
Broker stores a log; consumers track offsets
After consumption
Message gone
Retained until policy deletes it
Multiple independent consumers
Separate queues or subscriptions
Natural: each group reads independently
Replay old data
Generally not possible
Natural: rewind the offset
Ordering
Per queue or group, weaker with competing consumers
Per partition, strong
Per-message features
Delays, priorities, per-message retry, TTL common
Limited; retries and delays built by the consumer
Parallelism limit
Number of consumers
Number of partitions per group
Best for
Task distribution
Event distribution, pipelines, audit, replay
flowchart TD
S["New messaging need"] --> Q1{"Is each message a job done once and forgotten?"}
Q1 -->|"yes"| Queue["Use a queue with retries and DLQ"]
Q1 -->|"no"| Q2{"Do several systems need the same events, or replay, or per-entity order?"}
Q2 -->|"yes"| Stream["Use an event stream"]
Q2 -->|"no"| Q3{"Only a few thousand messages a day and one consumer?"}
Q3 -->|"yes"| Table["A database table or managed simple queue is enough"]
Q3 -->|"no"| Queue
Start at the top: jobs go to queues, shared facts and history go to streams, and small volumes may need neither. Do not adopt a streaming platform because it is fashionable.
NOTE
Retrying one failed event is harder on a log, because consumers read sequentially and a failing event can block its partition. Common patterns publish failures to a retry or dead-letter topic and move on, accepting out-of-order processing of those events.
Typical uses: activity tracking and analytics, event-driven integration (a service publishes OrderPlaced; billing, inventory and fraud subscribe independently), change data capture (database changes streamed to a search index, cache invalidation or warehouse, avoiding dual writes), stream processing (counts per minute, joins, anomaly detection), log aggregation and audit trails, and event sourcing. For partition sizing, lag and tuning, see Kafka scalability patterns.
Operating a stream: what to watch
Consumer lag: newest offset minus committed offset. Alert on lag, and better, on seconds behind real time.
Partition skew: uneven per-partition throughput signals hot keys.
Under-replicated partitions: copies that fell behind reduce durability and failover safety.
Disk and retention headroom: storage is write rate x retention x replication factor. For illustration, 20 MB per second kept 7 days with 3 replicas is roughly 20 x 86,400 x 7 x 3, about 36 TB. Compute with your own numbers.
Rebalance frequency: frequent rebalances point to unstable consumers (crashes, processing slower than the poll timeout).
In practice: clickstream and order events on Kafka
NOTE
A realistic scenario, not a case history. This is a composite example built from common production patterns. It does not describe any specific company, and every number in it is an illustrative assumption.
The situation
A marketplace has 2 million daily active shoppers, a 12-person data and platform group, and two consumers of behavior data that currently fight each other. Analytics wants clickstream (page views, searches, add-to-cart) in a warehouse within minutes. Risk wants order events scored for fraud within a few seconds, to hold suspicious orders before warehouse release. Today the web tier writes clicks to log files shipped nightly, and the order service calls the fraud scorer synchronously.
Illustrative pain: dashboards are a day old; the fraud scorer adds about 700 ms to checkout at p95 and its outage blocks orders; an analyst cannot replay last Tuesday's orders through a new fraud model. Peak load is roughly 8,000 click events per second and 40 order events per second (several events per order: placed, paid, shipped). The stack: Kafka (a 3-broker cluster, replication factor 3), a CDC connector such as Debezium reading the PostgreSQL write-ahead log, a stream processor for sessionisation, a columnar warehouse for analytics, Kubernetes, Prometheus and Grafana.
Starting architecture
Options considered
Option
What it fixes
What it costs
Risk
A. Smaller change: hourly batch plus an async queue to the fraud scorer
Fresher data, checkout no longer blocked
Low
No replay; analytics and risk still need separate pipelines
B. Kafka topics for clicks and orders, CDC for order changes, separate consumer groups for warehouse, sessionisation and fraud
One source of events, replay, independent consumers
Cluster and schema ownership; partition and key design
Operational learning curve; poor keys create hot partitions
C. Managed cloud queue and per-consumer subscriptions
Less to operate
Cheaper short term
Replay and per-key ordering weak; lock-in per consumer
The decision
ADR field
Content
Context
Two consumers need the same facts at different speeds, and history must be replayable.
Decision
Option B. Topic clicks keyed by session id (12 partitions, 3 days retention); topic orders keyed by order id (6 partitions, 14 days retention, schema-registered), fed by CDC from an outbox table so the event contract is not the raw table schema; a compacted order-state topic holding the latest state per order. Consumer groups: warehouse-sink, sessionizer, fraud-scorer. Fraud runs after the order commits; risky orders are placed on hold before warehouse release.
Consequences
Good: checkout no longer waits on fraud; a new fraud model can be validated by replaying 14 days of orders; each consumer scales and lags independently. Bad: fraud verdicts are asynchronous so a held-order state and a review queue are needed; the team now owns schemas, topics and a Kafka cluster.
Revisit when
Click volume doubles and 12 partitions cap the sessionizer, or retention cost per day exceeds the budget for replay value, or the team lacks capacity to operate Kafka and a managed service becomes the better trade.
Rollout and verification
The team would dual-publish for a period: the order service keeps its synchronous scorer call while the new fraud-scorer group runs in shadow, writing verdicts to a side table. Compare shadow verdicts with the synchronous ones for two weeks, then cut over behind a flag and remove the sync call. Run a load test at 3x peak click rate and replay a full day of orders through the consumer.
Dashboards: consumer lag in seconds per group, partition throughput skew, under-replicated partitions, scorer latency, verdict-to-release time, and rebalances per hour. Target: fraud lag under 5 seconds at p99 during peak; warehouse freshness under 5 minutes.
The 2 a.m. failure
A deploy of the fraud scorer adds a slow model call. Processing a batch now exceeds the poll timeout, so Kafka assumes the consumer is dead and rebalances; the group never settles and lag grows. The page reads: "fraud-scorer lag above 60 seconds". Orders sit in "held" state and the warehouse stops releasing them.
sequenceDiagram
participant K as Kafka
participant C1 as Scorer pod 1
participant C2 as Scorer pod 2
C1->>K: poll batch
Note over C1: slow model call, batch exceeds poll timeout
K->>K: member removed, rebalance
K-->>C2: partitions reassigned
C1->>K: commit rejected
Note over K: loop repeats, lag grows
First three checks: kafka-consumer-groups.sh --bootstrap-server <broker:9092> --describe --group fraud-scorer for lag per partition and member counts; the rebalance-rate panel and scorer logs for poll-timeout warnings; the deploy history to link the symptom to the model change. Mitigation: roll back the scorer, or reduce the batch size and raise the poll timeout; a documented policy releases low-value held orders after a time limit. Follow-up fix: make sure a batch always finishes inside the poll timeout (smaller batches, or keep polling while scoring runs by pausing the partitions and resuming when done), add a latency budget on the model call, and add lag-seconds alerting before a rebalance loop.
Architect's notes
The key is the design. Ask in review: what is the partition key, what is its cardinality, and what happens to the biggest key? A large share of the pain in Kafka designs traces back to this one decision.
Size partitions for next year's consumers, not today's. Partition count caps group parallelism and is painful to change; compute it from target consumer throughput with headroom.
Own the schema. Name an owner for each topic's contract and require compatibility checks in CI; an event nobody owns is a public API nobody maintains.
Cost trap: retention and replication multiply. Replay is cheap per day but the bill is write rate times retention times replicas; decide how many days of replay the business will actually use.
Who gets paged for lag? The consumer team, not the platform team. Put the lag alert in the consumer's runbook.
Defer exactly-once pipelines. Start with at-least-once and idempotent sinks; adopt transactions only for a pipeline that truly needs read-process-write atomicity.
Common mistakes
Expecting global ordering. Order exists only within a partition. Pick keys so that events needing order share a partition.
Choosing a poor key. A low-cardinality or skewed key creates hot partitions. Pick many distinct, evenly used values that match ordering needs.
Too few partitions at the start. Parallelism is capped by partition count, and changing it disturbs key mapping. Provision headroom.
Committing offsets before processing. A crash silently drops events. Process first, commit after, make handlers idempotent.
Retention shorter than the worst outage. A consumer down longer than retention loses events. Size for recovery time plus margin.
Treating "exactly-once" as end to end. It covers read-process-write inside the platform, not external calls. Use idempotency keys at the boundary.
Adopting a stream for a simple job queue. You pay complexity for features you do not use.
Key takeaways
A stream is a log, not a tray. Events are appended, retained by policy and read by position, so reading does not remove them.
Partitions give scale and ordering. Order is guaranteed within a partition only; throughput scales with partition count.
The key decides order and balance. Same key means same partition, so group related events and spread load evenly.
Consumer groups share work and isolate readers. Within a group each partition has one consumer; separate groups each see all events.
Offset commit timing is the delivery guarantee. Commit after processing for at-least-once, and add idempotency for effects outside the platform.
Replay and compaction are the superpowers. Rebuild state, fix bugs by reprocessing, and add new consumers over history.
Choose by workload. Jobs suit queues; shared facts, history and per-entity order suit streams.
Monitor lag. How far consumers trail the newest event is the key health signal.
Practice exercises
Exercise 1 — Starter (30 min): A topic has 6 partitions and a consumer group has 4 consumers. Draw a possible partition assignment. Then show what changes when a 5th, 6th and 7th consumer join, and when one consumer crashes. Note where each consumer resumes after the rebalance.
Exercise 2 — Intermediate (2 hours): For a ride-sharing app, design the topics for ride-requested, driver-location and ride-completed. For each, choose the key, the partition count with reasoning, the retention policy (time-based or compacted) and the consumer groups. Explain which topic risks hot partitions and how you would reduce it.
Exercise 3 — Advanced (half day): For the clickstream and fraud scenario, design a replay of 14 days of orders through a new fraud model without affecting live traffic: the topic or consumer group used, how verdicts are stored, how you compare old and new models, and how duplicates and out-of-order updates are handled. Add the lag alert you would set. Draw the data flow.
Lesson 7 of 14 · System Design Building Blocks: Data, Messaging and Scale
Understand log-based event streaming: topics, partitions, offsets, consumer groups, retention, compaction and replay, and when to choose a log over a message queue.
In the rest of this lesson
Ordering guarantees
Retention, compaction and replay
Streaming versus queue: how to choose
Operating a stream: what to watch
In practice: clickstream and order events on Kafka
Architect's notes
Enrolling is free. It unlocks every lesson in System Design Building Blocks: Data, Messaging and Scale, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.
Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.
Get new articles by email
Free. We email a confirmation link first, and you can unsubscribe from any email.