System Design: Distributed Transactions and Sagas

Sagas: a sequence of local transactions

A saga breaks a business transaction into steps. Each step is a local transaction in one service. Each step that changes state has a compensating action that semantically undoes it. If step N fails, the saga runs the compensations for steps N-1 down to 1.

For the order example:

Compensation is semantic, not a database rollback. A charge that was captured cannot be un-captured; you issue a refund, which the customer sees as two line items. The email you already sent cannot be unsent; you send a correction. Design the business process with this in mind.

Sagas give up isolation. Other requests can see the intermediate state: stock reserved but payment pending. You manage that with techniques such as:

• Semantic locks: mark records as PENDING so other flows know they are mid-saga.
• Commutative updates: design operations so order does not matter.
• Re-reading values before acting, and versioning records to detect concurrent changes.
• Reordering steps so the hardest-to-undo action comes last, and often placing a pivot step (the go/no-go point) after the compensatable steps, so that every step after it only needs to be retried until it succeeds.

Orchestration vs choreography

There are two ways to coordinate a saga.

A rough rule: choreography suits short flows with two or three services. As flows grow, orchestration usually pays for itself because someone can finally answer "where is order 4821 stuck?". Durable workflow engines persist the orchestrator's state and resume it after crashes. Replacing Cron and Spring Batch with Temporal shows one such engine handling retries and long-running state.

The transactional outbox pattern

Sagas rely on messages: "step done", "please charge". There is a trap. A service typically must do two things: update its database and publish a message. If it commits and then crashes before publishing, the next step never happens. If it publishes first and then fails to commit, other services act on something that never happened. This is the dual-write problem: two separate systems cannot be updated atomically by one process.

The transactional outbox removes it. In the same local database transaction as the business change, insert a row into an outbox table describing the message. A separate relay then reads the outbox and publishes to the broker, marking rows as sent.

Now both facts are committed or neither is. The relay may crash after publishing but before marking the row sent, so the message can be published more than once. That is acceptable only because consumers handle duplicates, which is the next section. The relay can be a polling process, or a change-data-capture tool reading the database log. Polling is simpler; log tailing reduces latency and load but adds infrastructure.

Order is only preserved if you keep it: publish outbox rows in insertion order per entity key, or include a version number so consumers can detect gaps.

Idempotency and "exactly-once effects"

Networks and brokers deliver messages at least once when reliability matters, so duplicates will happen. True exactly-once delivery across independent systems is not something you can assume. What you can build is exactly-once effects: the message may arrive several times, but its effect happens once. The recipe is at-least-once delivery plus idempotent processing, meaning repeating an operation has the same result as doing it once.

Practical techniques:

• Idempotency keys. The caller generates a unique key per logical operation (for example, a saga id plus step name). The receiver stores the key with the result in the same transaction as the business change. A repeated key returns the stored result without redoing the work.
• Inbox table / processed-message table. Consumers record message ids they have handled and skip duplicates, again in the same local transaction as their side effect.
• Naturally idempotent operations. "Set status to SHIPPED" is safe to repeat; "add 10 to balance" is not, unless guarded by a version check or key.
• Pass keys to external providers. Many payment APIs accept an idempotency key so a retried charge is not applied twice. Check your provider's documentation.

-- Same transaction as the business change
INSERT INTO processed_messages (message_id) VALUES ($1);
-- A unique constraint on message_id makes a duplicate fail the whole transaction
UPDATE accounts SET balance = balance - $2 WHERE id = $3;

If a duplicate arrives, the unique constraint violation aborts the transaction, so the debit is not applied twice. The consumer treats that specific error as "already processed" and acknowledges the message.

See Message Queues for delivery semantics in detail.

Try-Confirm/Cancel (TCC)

TCC is a pattern that sits between 2PC and sagas, and is common in payment and booking domains where resources can be held.

• Try: reserve the resource tentatively (hold funds, hold a seat) without finalising.
• Confirm: if all participants tried successfully, finalise each.
• Cancel: if any try failed, release the holds.

The reservations are first-class business concepts, so other users see "held" rather than a locked database row. Reservations normally carry an expiry so a crashed coordinator cannot hold inventory forever. Confirm and cancel must be idempotent, and a cancel that arrives before its try (due to reordering or timeouts) must be handled by recording it so the late try is rejected.

TCC gives stronger isolation than a plain saga, but each participant must implement three operations and a reservation model, which is real design work.

Failure scenarios to walk through

A good way to test a design is to take each step and ask what happens if it fails or repeats. For the order saga with an orchestrator:

Two rules fall out of the table:

• Compensations must be retried until they succeed, so they must be idempotent and as simple as possible. A compensation that can fail permanently needs a manual-repair queue.
• Every saga needs a timeout and an owner. A saga stuck forever in PENDING is a data-quality incident waiting to be noticed. Track age, and alert.

In practice: a flight, hotel and payment booking saga

The situation

An online travel agency sells flight-plus-hotel bundles. Eighteen engineers run Spring Boot services on Kubernetes, each with its own PostgreSQL database, with Kafka for events. Flights and hotels come from third-party supplier APIs, and card payments go through a payment gateway. The suppliers do not join any commit protocol, flight fares and hotel rooms can each be held for about 15 minutes, and payment supports authorise now and capture later. Sales peaks reach roughly 40 bookings per second (assumption).

Today a single Booking API request calls the flight supplier, the hotel supplier and the gateway in sequence, with try/catch blocks that attempt to undo earlier steps. The numbers that hurt (all illustrative): about 2 percent of bookings end inconsistent (paid but not ticketed, or ticketed but not paid), support needs around 25 minutes to fix each, and a supplier timeout leaves nobody certain what happened.

Starting architecture

There is no durable record of progress. If the pod dies after step 2, nothing knows steps 1 and 2 happened.

Options considered

The decision

Rollout and verification

The pivot is ticketing. States above it can compensate; states below it only move forward. Roll out behind a feature flag: first internal bookings against supplier sandboxes, then 1, 5 and 25 percent of traffic, with the old flow as the instant fallback. Keep the 2 percent inconsistency rate as the comparison baseline. Dashboards and alerts: saga age percentiles, compensation rate per supplier, depth of the manual-repair queue, duplicate-key hits (proof idempotency is working), and the gap between captured payments and ticketed flights, which should be zero.

The 2 a.m. failure

The page reads: "Saga compensation backlog above 50 for 10 minutes". The hotel supplier is returning 5xx errors, so release calls for cancelled bookings keep failing.

Detection comes from the saga_compensation_pending gauge and a retry-count alert. First three checks: filter failed activities by type ReleaseHotelHold in the workflow UI, check the supplier's status and the circuit breaker state, and confirm the retries reuse the same idempotency key. Mitigation: customers are safe, because payment is already voided and the hold expires by itself. Pause new bundles that use that supplier, and let retries continue. Follow-up fix: a per-supplier circuit breaker, a manual-repair queue for holds that outlive expiry, and a post-incident check that no retry ever used a fresh key.

Architect's notes

• Order steps by reversibility. Put the hardest-to-undo action last among the steps that might fail, and write down which one is the pivot.
• Prefer holds with expiry. An expiring hold is a safety net that works even if your orchestrator is down.
• Ask in the review: "What does the customer see at each state, and what does support see?" Intermediate states are a product decision, not just a technical one.
• Cost trap. Workflow history grows with every step and retry. Plan retention and payload size early.
• Ownership. Name one team that owns the orchestrator and its stuck-saga queue. Service teams own their own compensations.
• Defer on purpose. Do not add cross-supplier optimisation or partial-refund logic until the basic saga is boring.

Common mistakes

• Dual writes to a database and a broker. A crash between the two leaves them disagreeing. Use the transactional outbox or change data capture.
• Assuming exactly-once delivery. Duplicates can and will arrive. Make every consumer idempotent and store processed ids with the side effect.
• Treating compensation as rollback. Real-world actions cannot be undone, only offset. Design visible, business-meaningful compensations and order steps so the irreversible one comes last.
• Using 2PC across unreliable or external systems. Third-party APIs and many brokers cannot participate, and the blocking risk spreads. Prefer sagas or a redesigned boundary.
• Ignoring intermediate states. Other requests see half-finished sagas. Use status fields and semantic locks, and show users honest states like "processing".
• No timeouts or monitoring on sagas. Stuck workflows silently hold stock and money. Persist saga state, set deadlines, and alert on age.
• Hiding business logic in a choreographed event web. Nobody can see the flow. Switch to orchestration when the number of steps or services grows.

Key takeaways

• Atomicity across services is a protocol, not a feature. A single database transaction cannot cover several service-owned databases.
• A timeout means "unknown". Designs must handle the ambiguous outcome, not just success and failure.
• Two-phase commit blocks. Participants hold locks after voting yes and wait for the coordinator, so it hurts availability and throughput.
• Sagas trade isolation for availability. Local transactions plus compensations give eventual consistency, with visible intermediate states.
• Choose orchestration as flows grow. Choreography is loose but implicit; orchestration is visible and easier to operate.
• The outbox solves dual writes. Write the message with the data in one local transaction, then relay it, accepting at-least-once publication.
• Aim for exactly-once effects. At-least-once delivery plus idempotent processing, using keys and processed-message tables.
• Redraw boundaries first. If two services must always change together, consider merging them before building a protocol.

Practice exercises

Exercise 1 — Starter (30 min): Write the saga table (step, local transaction, compensation) for a trip booking that reserves a flight, a hotel and a car. Mark which step should be the pivot (the point of no return) and justify the order.

Exercise 2 — Intermediate (2 hours): Draw a sequence diagram of the order saga using an orchestrator and the outbox pattern. Then trace three failures on it: the payment service crashes after charging but before replying; the relay publishes a message twice; the orchestrator restarts mid-saga. For each, state what the database contains afterwards and which mechanism restores consistency.

Exercise 3 — Advanced (half day): Design a money-transfer feature between accounts held in two different services. Compare a 2PC design, a TCC design with holds that expire, and a saga with compensations. For each, specify the failure behaviour during a 10-minute outage of one service, what customers can see, the idempotency scheme, and the monitoring you would add. Recommend one and write the trade-off in three sentences.

Related reading

• Message Queues - delivery semantics, acknowledgements and idempotent consumers that sagas depend on.
• Replacing Cron and Spring Batch with Temporal - a durable workflow engine in practice, useful for saga orchestration.
• CQRS and Event Sourcing - an alternative way to model state changes as a reliable stream of events.
• Architecture Styles - why service boundaries decide how many distributed transactions you need.