System Design: Message Queues and Asynchronous Work

Validation errors go straight to the DLQ because retrying cannot help; timeouts and 5xx responses deserve backoff and retry.

Delivery semantics, stated precisely

• At-most-once. Zero or one delivery. The queue marks it delivered before the consumer finishes; a crash loses the message. Acceptable for disposable data such as a non-critical metric sample.
• At-least-once. One or more deliveries. The message is removed only after an ack, so a crash or lost ack causes redelivery. This is the default of most practical queues and the model to assume.
• Exactly-once. Be careful with this phrase. Over an unreliable network a sender cannot tell whether a missing acknowledgement means the message was not received or the ack was dropped, so "exactly-once delivery" between independent systems is not achievable in general. What is achievable is exactly-once effect: at-least-once delivery plus idempotent processing, or transactional processing inside one system where reading input, updating state and writing output commit atomically. The latter does not extend automatically to an email provider or payment gateway on the other side.

Idempotent consumers

An operation is idempotent if doing it twice has the same effect as once. "Set order 123 to shipped" is; "add 10 to the balance" is not. Techniques: natural idempotency (set, do not increment); a deduplication key assigned by the producer (not generated per send attempt), recorded by the consumer in the same database transaction as the business effect; conditional writes with unique constraints; and passing the id downstream as an idempotency key to external services, as in API design.

begin transaction
  if exists(processed_ids, message.id): commit; ack; return
  apply business change
  insert into processed_ids(message.id)
commit
ack

If the consumer crashes after the commit but before the ack, the redelivered message finds its id and is skipped.

Ordering

A single queue with a single consumer processes in order but caps throughput at one consumer. With competing consumers, completion order can differ from arrival order, and retries make it worse: a failed message returns after newer ones were processed. If you need order only per entity (one order, one account), route all that entity's messages to the same consumer via a partition key or message group; different entities still run in parallel, usually at some throughput cost. Strict global order is a throughput ceiling; question whether you need it. A sturdier approach is tolerance: carry a version number and ignore messages older than what the entity has already seen.

Back-pressure: protecting the slow side

A queue hides a speed mismatch but does not fix it. If producers add 1,000 messages per second and consumers finish 600, the queue grows by 400 per second; for illustration, after an hour that is 1.44 million waiting messages, each getting older. Back-pressure pushes the problem back toward the producer or sheds load before things break: alert on queue depth and age of the oldest message (age measures how late work is), autoscale consumers up to what downstream systems can handle, bound the queue and make producers reject or degrade when full, rate limit at the entry point (rate limiting), give messages a time-to-live so stale jobs are dropped, and limit concurrency per downstream so more consumers do not just overload the database or API they call.

The work-queue pattern in one picture is photo upload: the API stores the original in object storage, enqueues {photoId, objectKey}, returns 202 Accepted; workers make thumbnails, update status and ack; bad images end in the DLQ.

One subtle pitfall is the dual write: saving to the database and publishing to the queue are two operations, and a crash between them leaves the system inconsistent. The usual fix is the transactional outbox, covered in distributed transactions.

In practice: order fulfilment with invoice, email and warehouse messages

The situation

A home-goods retailer ships roughly 30,000 orders a day, and about 3 times that during seasonal sales. The order service is a Java application on Kubernetes with PostgreSQL. On "order placed" it does, inside the HTTP request: create the invoice via a finance API, send a confirmation email through an SMTP provider, and push a pick request to a warehouse partner's HTTP API. A team of nine owns all of it.

The metrics that hurt (illustrative): checkout p99 is 6 seconds on sale days because three downstream calls run in line; when the warehouse API is down, checkout returns errors even though the order and payment succeeded; and a retry bug sometimes sent customers two confirmation emails.

Starting architecture

Checkout latency is the sum of three third-party calls, and its availability is their product.

Options considered

The decision

Rollout and verification

Ship the outbox and relay first with no consumers reading, and verify rows publish. Then enable the email consumer in shadow mode (log only), then live for 5 percent of orders behind a flag, then the invoice and warehouse consumers. Run a load test at 3x sale volume including a killed consumer pod and a paused warehouse stub. Dashboards: oldest-message age per queue, queue depth, publish-to-ack latency, DLQ depth, redelivery rate, duplicate-suppressed count from the dedup table, and checkout p99. The target: age of the oldest message in each queue stays below its SLO (say 2 minutes for email, 15 for warehouse) at peak.

The 2 a.m. failure

The warehouse partner's API starts returning 503. The page reads: "warehouse queue oldest message age above 15 minutes". Messages are retried with backoff, depth grows, and with 5 quick attempts a burst of orders would have landed in the DLQ even though the outage is transient.

The first three checks: rabbitmqctl list_queues name messages_ready messages_unacknowledged to see depth and in-flight counts; the worker logs and error-rate panel to confirm the 503s are the partner's; and the DLQ depth, to see whether retries are being exhausted. Mitigation: pause the consumer (or open its circuit breaker) so messages stay safely in the queue rather than burning their attempts, notify the partner, redrive anything already dead-lettered once the partner recovers, and resume with limited concurrency. Follow-up fix: retry counts should be per time window, not just per attempt, so a long dependency outage never exhausts attempts; add an explicit "dependency down" path that holds messages.

Architect's notes

• Alert on age first, depth second. A deep queue draining fast is fine; a shallow queue whose oldest message is old is not. Define a freshness SLO per queue.
• Ask who owns each DLQ. If the answer is "the platform team", no one will read it. Each DLQ needs an owning team, an alert and a weekly review.
• Separate transient from permanent errors in code review. The classification in the handler is the single biggest factor in whether retries help or hurt.
• Check the idempotency boundary of every external call. If a vendor offers no idempotency key, a duplicate effect is possible and needs a compensating check.
• Count the broker in the availability budget. Moving work off the request path raises checkout availability but adds a dependency; keep the outbox so publishing can lag without losing orders.
• Defer fancy ordering. Start with per-order partitioning only where a real ordering bug exists; global FIFO is rarely worth its throughput ceiling.

Common mistakes

• Assuming exactly-once delivery. Duplicates will happen. Make consumers idempotent and use dedup keys.
• Retrying forever. One poison message can burn capacity indefinitely. Cap attempts, back off with jitter, dead-letter.
• Ignoring the DLQ. Messages pile up unseen and customers lose emails or orders. Alert on depth and build a redrive tool.
• Visibility timeout shorter than processing time. The queue hands the message to a second worker. Set it above the longest normal run or extend the lease.
• Assuming FIFO means ordered processing. Partition by entity key and make handlers tolerate stale messages.
• Unbounded queues without alerts. A growing backlog hides a capacity problem until it is an outage. Alert on oldest age.
• Using a queue for work the user is waiting on. Provide status tracking or do that step synchronously.

Key takeaways

• A queue decouples time, scale and failure. Producers get a fast acknowledgement; consumers work later and independently.
• Queue for tasks, topic for events. Point-to-point shares work; publish/subscribe copies each event to every subscriber.
• Ack after the work is done. Acknowledgements and visibility timeouts stop a crash from losing a message.
• Assume at-least-once. Exactly-once delivery between independent systems is not achievable; build exactly-once effect with idempotent consumers.
• Cap retries and keep a dead-letter queue. Backoff with jitter handles transient errors; a monitored DLQ catches poison messages.
• Ordering is per key, not global. Partition by entity to keep related messages in sequence while preserving parallelism.
• Watch the age of the oldest message. It is the clearest signal that consumers cannot keep up.
• Back-pressure is a design requirement. Bound queues, limit concurrency, scale consumers and shed stale work.

Practice exercises

Exercise 1 — Starter (30 min): List eight operations in an online store (checkout payment, order confirmation email, inventory decrement, invoice PDF, recommendation update, fraud check, shipping label, analytics event). Mark each synchronous or asynchronous with a one-line reason, and say which would use a queue and which a topic.

Exercise 2 — Intermediate (2 hours): Write the pseudocode for an idempotent email-sending consumer, including the deduplication table, what is committed together, the order of "send", "record" and "ack", and what happens if the process crashes between each pair of steps. Identify the step where a duplicate email could still occur and explain what an idempotency key at the email provider would change.

Exercise 3 — Advanced (half day): Design the order-processing pipeline for a flash sale receiving about 20,000 orders per second for ten minutes (illustrative) while the payment provider handles at most 2,000 per second. Specify the queue topology, partitioning key, consumer concurrency, retry and DLQ policy, how users learn the outcome, when you reject new orders, and the dashboard and alerts. Draw it with Mermaid or on paper.

Related reading

• System Design: Event Streaming and Logs - the log-based alternative to queues, with replay and consumer groups.
• System Design: Reliability Basics - retries, backoff, jitter and idempotency, the ideas queues rely on.
• System Design: Distributed Transactions - the transactional outbox and sagas for keeping a database and a queue consistent.
• System Design: Rate Limiting - controlling the rate at which work enters the system.
• Designing a Notification Service - a worked example of queues, retries and fan-out in practice.