QuickCart Stage 5: The Order Pipeline With Queues, Events and Sagas

Delivery guarantees

• At-most-once: zero or one delivery. The message is marked done before the work, so a crash loses it. Fit only for data you can afford to lose.
• At-least-once: one or more deliveries. It is marked done after the work, so a crash or lost acknowledgement causes redelivery. Assume this by default.
• Exactly-once: one effect. Between independent systems over a network, a sender cannot tell a lost message from a lost acknowledgement, so exactly-once delivery cannot be assumed. What can be built is exactly-once effect: at-least-once delivery plus idempotent processing, or an atomic transaction inside one system.

In Kafka, "marked done" is committing the consumer's offset. Commit before processing and a crash skips events; process, then commit, and a crash re-reads them. Kafka also has two features often called exactly-once, each with a limited scope:

QuickCart's orchestrator writes to PostgreSQL and calls queues and a gateway, so Kafka transactions do not cover it. The rule for the pipeline: every hop is at-least-once, and every consumer is idempotent.

The first hop (API to shard) is one local transaction. The relay can republish, so every event carries a unique event_id. Queue workers get the order id as a business key, and the payment worker sends an idempotency key to the gateway (Stage 6).

The consumer stores a record of handled events in the same transaction as its change; a unique key makes a duplicate harmless:

CREATE TABLE inbox (
  consumer   text        NOT NULL,
  event_id   uuid        NOT NULL,
  handled_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer, event_id)
);

BEGIN;
INSERT INTO inbox (consumer, event_id) VALUES ('orchestrator', $1)
ON CONFLICT DO NOTHING;
-- If that inserted 0 rows, the event was already handled: COMMIT and acknowledge.
UPDATE orders SET status = 'STOCK_RESERVED'
WHERE order_id = $2 AND status = 'PENDING';
COMMIT;

If the consumer crashes after the commit but before acknowledging, the redelivered event finds its inbox row and does nothing. The status = 'PENDING' guard also stops a late, old event from moving an order backwards.

Dead-letter queues and retries

A transient failure (the gateway returned 503) usually succeeds on retry. A permanent failure (a malformed message) never does. A message that always fails is a poison message; retried forever, it burns capacity and blocks others. The standard design:

1. Classify the error in the handler. Permanent errors go straight to the dead-letter queue; timeouts and 5xx responses are retried.
2. Retry with exponential backoff and jitter. The wait is a random value up to a ceiling that doubles each attempt and is capped. With a 1-second base and a 60-second cap (illustrative) the ceilings are 2, 4, 8, 16 and 32 seconds. The randomness stops failed messages retrying together; see reliability basics.
3. Limit attempts, then dead-letter. The message moves to a dead-letter queue (DLQ). Managed queues express the limit their own way: Amazon SQS uses a redrive policy with a maxReceiveCount, and advises a longer retention on the DLQ than on the source queue.
4. Alert and redrive. Alert on DLQ depth, because an unwatched DLQ is where orders are forgotten. A redrive tool moves messages back to the source after the fix.

Figure 5. What a worker does with one message

How to read this: only transient errors are retried. A permanent error skips the retries, because retrying cannot help.

On a log, Kafka's consumer groups have no per-message DLQ in the broker (Kafka Connect and newer share groups add their own forms), and a failing event blocks its partition. The usual pattern is for the consumer to publish the failed event to a retry or dead-letter topic and move on, accepting out-of-order handling; that is why guarded updates matter. If a downstream is down for long, pause consumption rather than burn attempts.

Writing the order and the event together

The transactional outbox pattern

Checkout must save the order and announce it, in two systems: the orders database and the broker. This is the dual-write problem, and no ordering is safe. Save, then publish: a crash in between leaves an order nobody processes. Publish, then save: a crash or a failed save announces an order that does not exist.

The transactional outbox writes the message into the same database as the business change. One local transaction inserts the order and an outbox row describing the event; a separate relay later publishes unpublished rows and marks them sent.

CREATE TABLE outbox (
  seq          bigserial   PRIMARY KEY,
  event_id     uuid        NOT NULL UNIQUE,
  aggregate_id text        NOT NULL,
  event_type   text        NOT NULL,
  payload      jsonb       NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now(),
  published_at timestamptz
);
CREATE INDEX outbox_unpublished ON outbox (seq) WHERE published_at IS NULL;

BEGIN;
INSERT INTO orders (order_id, shopper_id, store_id, status, placed_at)
VALUES ($1, $2, $3, 'PENDING', now());
INSERT INTO outbox (event_id, aggregate_id, event_type, payload)
VALUES ($4, $1, 'OrderPlaced', $5::jsonb);
COMMIT;

Either both rows exist or neither does. The relay loops: select the oldest unpublished rows, publish each to Kafka with the order id as key, wait for the acknowledgement, and set published_at.

Figure 6. Checkout writes the order and the outbox row together

How to read this: the shopper's 202 depends only on the first four steps. If the relay crashes before the last step, the row is simply published again.

Four points make it correct in practice:

• The outbox sits on the same shard as the order. A transaction cannot span shards (Stage 4), so each shard has its own outbox and relay.
• Publication is at-least-once. A relay that crashes after publishing republishes, so consumers deduplicate by event_id.
• Run one active relay per shard, since two relays polling the same rows could publish out of order. It is a leader-run process (see background jobs).
• Order is per aggregate and not perfect: sequence numbers can commit out of order, so carry a version in the payload if consumers must detect gaps. A scheduled job deletes old published rows.

Change data capture (reading the write-ahead log) is the alternative to polling: lower latency, more infrastructure. Polling suffices at 58 checkouts per second. See distributed transactions.

Coordinating the steps

The saga pattern

Stock, payment and courier have different owners and data, and a two-phase commit would block on the external gateway. A saga replaces the one big transaction with a sequence of local transactions, one per step. Each step has a compensating action that semantically undoes it. If step N fails, the saga runs the compensations for steps N-1 down to 1.

Compensation is semantic, not a rollback: a void is a new, visible action, and it can fail too. So it must be idempotent and retried until it succeeds, and a compensation that keeps failing goes to a manual-repair queue with an owner.

Orchestration or choreography? In orchestration a central orchestrator tells each service what to do and records each order's state: the flow is visible in one place, but the orchestrator must be reliable and free of business rules. In choreography services react to each other's events (StockReserved triggers payment): no controller, but the flow is implicit across services and hard to debug. Choreography suits two or three steps; past that, orchestration usually pays for itself.

QuickCart uses orchestration, because it needs to answer "where is order 9001 stuck?" in seconds. The orchestrator is a consumer group on order-events. For each event it loads the order, applies the next transition with a guarded update, and sends the next command to a queue. Workers reply with events published through their own outbox.

Figure 7. The life of an order as a state machine

How to read this: states above CONFIRMED can still be undone, and each failure arrow leads to compensation. CONFIRMED is the point of no return. NEEDS_REVIEW is where a human takes over, because compensations can fail.

A saga gives up isolation: other readers see half-done orders. QuickCart makes the states honest ("Placing your order") and uses holds with expiry so a stalled saga cannot lock stock forever. Every saga has a deadline: an order stuck in a non-final state past it is a defect that a scheduled job finds. The payment side is in Stage 6.

Running work in the background

Background jobs and schedulers

A background job runs outside a request. QuickCart has three kinds: cron-like jobs on a schedule (purge published outbox rows, find stuck orders, later reconcile payments); delayed jobs set by an event for a future time ("if order o-7001 is still PENDING after 10 minutes, illustratively, cancel it"); and retried jobs rescheduled with backoff.

Jobs must be idempotent. Kubernetes' CronJob documentation says a CronJob creates a Job approximately once per scheduled time, that two Jobs might be created or none, and that Jobs should therefore be idempotent. "Cancel orders still PENDING older than 10 minutes" is idempotent, since a second run finds nothing to do. "Add 10 loyalty points" is not, until it carries a deduplication key.

A leader-run task must have one runner. The outbox relay and the stuck-order scanner must not run on every instance at once. Options: a CronJob with concurrencyPolicy: Forbid (skip a run if the previous is still going; Allow is the default), a lease row in the database that one instance holds and renews, or a leader chosen by a consensus service (consensus and leader election). A lease can expire while its holder works, so the job must still tolerate a second runner.

QuickCart keeps delayed jobs in a table, claimed with SKIP LOCKED. PostgreSQL's documentation notes that skipping locked rows gives an inconsistent view, unsuitable for general work but fine for queue-like tables with several consumers:

CREATE TABLE jobs (
  job_id       uuid        PRIMARY KEY,
  job_type     text        NOT NULL,
  dedup_key    text        NOT NULL UNIQUE,
  payload      jsonb       NOT NULL,
  run_at       timestamptz NOT NULL,
  status       text        NOT NULL DEFAULT 'queued',
  attempts     int         NOT NULL DEFAULT 0,
  locked_until timestamptz
);

-- Claim up to 20 due jobs; a crashed worker's lease expires after 60 seconds
UPDATE jobs
SET status = 'running', attempts = attempts + 1,
    locked_until = now() + interval '60 seconds'
WHERE job_id IN (
  SELECT job_id FROM jobs
  WHERE run_at <= now()
    AND (status = 'queued' OR (status = 'running' AND locked_until < now()))
  ORDER BY run_at
  LIMIT 20
  FOR UPDATE SKIP LOCKED
)
RETURNING job_id, job_type, payload;

When checkout writes order o-7001 it also inserts a job with dedup_key = 'expire-order:o-7001' and run_at = now() + interval '10 minutes'; with ON CONFLICT (dedup_key) DO NOTHING, inserting it twice is harmless. When it runs, it sends CancelOrder to the orchestrator, which ignores it if the order is already final. At this volume a table is enough.

Staying alive under load

Backpressure

A queue hides a speed mismatch; it does not remove it. If messages arrive faster than consumers finish them, the backlog and the waiting time grow. Backpressure pushes that pressure back toward the source or sheds load before things break:

• Bounded queues. Give each queue a maximum depth. An unbounded queue turns overload into a slow outage of stale work.
• Load shedding. When the queue is full, refuse new work early and clearly (503 with Retry-After, or "high demand, try again shortly"), least valuable work first. Never silently drop work you already acknowledged.
• Lag and age as signals. Alert on the age of the oldest message and on consumer lag, ideally in seconds. Depth alone misleads: a deep queue draining fast is fine; a shallow one with an old message is not.

A worked example, with illustrative numbers. The gateway contract allows 40 authorisations per second, and a promotion pushes checkouts to the 58 per second peak for 10 minutes. The surplus is 58 - 40 = 18 per second. Over 600 seconds the backlog reaches 18 x 600 = 10,800, and draining it at 40 per second takes another 270 seconds if nothing new arrives, so the last shopper waits over four minutes. If shoppers tolerate about 60 seconds, the payment queue should hold at most 40 x 60 = 2,400 messages. At 18 extra per second it fills in about 133 seconds, and checkout then sheds the excess with a clear message.

Shedding is the safety net, not the plan: here about 8,400 checkouts would be turned away, so the real fix is a higher limit or a second provider. Kafka does not slow producers because consumers lag, so the limit belongs at the checkout API (rate limiting).

One checkout, end to end

Order o-7001 goes through the whole pipeline, with a failure: no courier is free.

Figure 8. One checkout with a failure and its compensation

How to read this: the shopper's part ends at the 202. Every arrow to a service is a retryable command and every reply an event; the compensations run in reverse order of the steps that succeeded.

1. The API writes the order (PENDING) and the outbox row in one transaction and returns 202.
2. The relay publishes OrderPlaced (key o-7001); the orchestrator reads it and records its inbox row.
3. ReserveStock then AuthorisePayment go through queues; each worker replies with an event.
4. AssignCourier fails after retries within the assignment window (illustratively 2 minutes), so the order moves to COMPENSATING.
5. VoidAuthorisation and ReleaseStock are retried until they succeed; the order becomes CANCELLED and OrderCancelled is published.
6. The notifications group tells the shopper. Nothing is charged or held.

Had a courier been found, the order would reach CONFIRMED, and OrderConfirmed would trigger the confirmation message. That message goes last because it cannot be undone.

Decisions and trade-offs

In practice: QuickCart at this stage

This is a realistic scenario, not a case history. It is an educational reference design, and every number is an illustrative assumption.

The situation

QuickCart is at about 5 million monthly active users, heading to 10 million. Checkout is still the chain of Figure 1. The lunchtime peak is about 30 checkouts per second and p99 checkout latency is near 5 seconds. One evening a slow gateway caused a wave of timeouts and retries; support found 40 orders paid twice in one hour (illustrative).

Starting architecture

Figure 9. The starting architecture: checkout calls five dependencies in line

How to read this: every arrow is a synchronous call inside one request, made in the numbered order. The two dashed boxes are outside QuickCart's control, and the request is as slow and fragile as the worst of the five.

Options considered

The decision

Rollout and verification

Ship behind a feature flag. First add the outbox and relay and verify rows reach Kafka. Then run the orchestrator in shadow mode, logging what it would do while the synchronous path still does the work, and compare for a week. Move 1, 5 and 25 percent of checkouts, then all, keeping the synchronous path as a fallback. Rehearse in staging: kill the relay, kill an orchestrator mid-saga, pause the gateway stub, push a malformed message to the DLQ. The outcome must match Figure 8: no double charges, no stuck holds.

Dashboards: time from 202 to CONFIRMED, orders by state, oldest outbox row, orchestrator lag, queue age, DLQ depth and the shed rate.

The 2 a.m. failure

A deploy at 01:50 leaves the outbox relay on one shard crash-looping. Orders are accepted and shoppers get 202, but no OrderPlaced is published, so the orchestrator never sees them. The page reads "oldest unpublished outbox row older than 30 seconds on shard 7".

First three checks: the relay's pod status and logs; the backlog on the shard (SELECT count(*), min(created_at) FROM outbox WHERE published_at IS NULL;); and whether other relays can publish to Kafka, to separate a relay bug from a broker problem.

No order is lost, since each is committed with its outbox row. Mitigation: roll back or restart the relay, throttling its publish rate if the backlog is large; tell support the orders are delayed, not lost.

A second hazard: the 10-minute expiry job may fire before the relay recovers. When the late OrderPlaced arrives for an order already cancelled, the guarded update (WHERE status = 'PENDING') finds nothing to do and the inbox records the event, so the cancelled order is not revived. Follow-ups: alert on outbox age well before the expiry window and set the expiry longer than the worst-case publish delay plus the saga.

Architecture after this stage

Figure 10. QuickCart after Stage 5: the order pipeline added

How to read this: the sync arrow ends at the orders shards, and that is the shopper's whole wait. Everything in the zone is asynchronous. The earlier stages' read path (load balancer, cache, replicas, CDN) is unchanged and not repeated.

What QuickCart looks like now

Architect's notes

• Draw the sync line first. Decide what the shopper must see before the response and what cannot be undone.
• Treat every consumer as a retry target. If it cannot run twice with the same message and give the same result, it is not ready.
• Defer what the volume does not need. At about 58 checkouts per second, a polling outbox, a jobs table and a plain orchestrator are enough.

Common mistakes

• Dual writes to a database and a broker. Use the outbox.
• Assuming exactly-once delivery. Kafka's idempotent producer is per session and partition, and its transactions stop at Kafka's edge. Make consumers idempotent.
• Retrying forever, or without jitter. One poison message holds a worker; synchronised retries hammer a service.
• Unwatched dead-letter queues. Orders sit unnoticed. Alert, and give each DLQ an owner.
• Treating compensation as rollback. A void can fail. Make it idempotent, retry it, and give failures a repair queue.

Key takeaways

• Keep synchronous only what the shopper must see or the system must verify. Checkout accepts the order and returns 202.
• Queues carry commands, logs carry events. A queue gives each task to one worker; a topic gives each event to every group.
• Assume at-least-once everywhere. Kafka's idempotent producer (per session) and transactions (inside Kafka) are limited; consumer idempotency is the general answer.
• The outbox solves the dual write. The order and its event commit together; the relay publishes later, at-least-once.
• A saga trades isolation for availability. Local transactions plus idempotent compensations, honest states, a deadline per order.
• Jobs must be idempotent, and bounded queues keep overload visible.

Practice exercises

Exercise 1 - Starter (30 min): Draw the state machine for an order whose payment is declined at step 2. List the states, the compensation that runs and the notification sent. Mark the point of no return and say why.

Exercise 2 - Intermediate (1 hour): QuickCart now publishes 8 events of 1.5 KB per order, with 14 days of retention and 3 replicas. Compute the peak events per second and the disk needed at 500,000 orders per day.

Exercise 3 - Advanced (half day): Stores can reject an order. Add a "store accepts" step with a 2-minute timeout to the saga. Decide where it goes, which step becomes the pivot, each compensation, and how the jobs table implements the timeout. Update the state diagram.

Related reading

• QuickCart Stage 4: Scaling Writes - the sharded orders database that holds each outbox.
• QuickCart Stage 6: Payments and the Ledger - idempotency keys, the ledger and reconciliation.
• QuickCart Stage 7: Search, Notifications, Uploads and Analytics - new consumers of order-events.
• Message Queues and Asynchronous Work - acknowledgements, visibility and ordering.
• Event Streaming and Logs - partitions, groups, compaction and replay.
• Distributed Transactions and Sagas - two-phase commit, TCC and the outbox in general.

Official documentation

• Apache Kafka: design, delivery semantics, log compaction and consumer groups
• Apache Kafka: producer configuration (enable.idempotence, transactional.id)
• KIP-98: Exactly Once Delivery and Transactional Messaging
• PostgreSQL: the locking clause of SELECT, including SKIP LOCKED
• Kubernetes: CronJob, concurrency policy and idempotent jobs
• Amazon SQS: dead-letter queues and redrive