What you'll learn: By the end of this lesson you will be able to list the eight fallacies of distributed computing, explain for each one what is true instead, say why a timeout tells you nothing about whether a remote call worked, and turn each fallacy into a design habit you can check in a review.
A person posts a letter and, a week later, hears nothing back. Did the letter get lost? Was it delivered and ignored? Did the reply get lost on the way home? Nobody can tell from the silence alone. A program that calls another machine over a network is in the same position every time the reply is late, and most of the bugs in distributed systems come from writing code as if the post office were infallible.
Peter Deutsch and colleagues at Sun Microsystems wrote the assumptions down as a list. The page that records it opens: "Essentially everyone, when they first build a distributed application, makes the following eight assumptions." This lesson takes each one, says what is true instead, and ends with a worked ShopStream scenario. The related lessons, Resilience patterns and CAP Theorem and PACELC, go deep on the cures.
Read from the sources listed at the end and checked on 2026-11-08; no product versions are involved.
The core idea
A remote call is not a local call that happens to be slow. It has a different set of outcomes. Waldo, Wyant, Wollrath and Kendall made the argument in "A Note on Distributed Computing" (Sun Labs, 1994). They name latency, memory access, partial failure and concurrency as the differences between local and distributed computing, and this lesson leans on partial failure. When a local component fails you can know the state of the system before and after; no such determination can be made in a distributed system. So interfaces between components must say, whenever possible, what caused a failure, and must allow a reasonable state to be reconstructed when the cause cannot be determined.
Three examples run through the lesson, each labelled: a minimal example (the timeout that tells you nothing), a realistic example (ShopStream's checkout calling a payment provider) and a failure example (retries multiplying across layers).
The eight fallacies, and what is true instead
The list was started by L. Peter Deutsch in 1994 with seven items, drawing on four that Bill Joy and Dave Lyon had identified, and James Gosling added the eighth around 1997, according to the Wikipedia article on the list. Deutsch later suggested a ninth, that the party you are talking to is trustworthy.
| # | Fallacy | What is true | Design habit |
|---|---|---|---|
| 1 | The network is reliable | Messages are lost or delayed, and not only by the wire | Timeouts, bounded retries, idempotent operations |
| 2 | Latency is zero | Every call takes time, and it varies | Fewer round trips, deadlines, caching, asynchronous work |
| 3 | Bandwidth is infinite | Payload size has a cost, especially in bulk | Paginate, send identifiers not whole objects, compress |
| 4 | The network is secure | Nothing is trusted because of where it sits | Encrypt in transit, authenticate every caller |
| 5 | Topology doesn't change | Servers, routes and addresses come and go | Discovery by name, health checks, no hard-coded addresses |
| 6 | There is one administrator | Many teams and vendors own parts of the path | Explicit contracts, versioning, shared observability |
| 7 | Transport cost is zero | Serialising, copying and moving data costs CPU and money | Count cross-zone and cross-region traffic in the design |
| 8 | The network is homogeneous | Different machines, languages and versions interoperate | Standard formats, tolerant readers, no shared binary assumptions |
Fallacy 1: the network is reliable
Peter Bailis and Kyle Kingsbury's survey "The Network Is Reliable" collected published reports of failures. The point is not that networks are always down, but that failures are common enough to design for. They cite a Microsoft data-centre study in which, on average, 5.2 devices and 40.8 links failed per day, and they note that "not all partitions originate in the physical network": crashes, race conditions and scheduler delays can drop or delay messages too. Their advice is to consider partition risk at design time, when decisions are cheap, and not during an outage.
Fallacy 2: latency is zero
Each remote call adds a delay that varies from call to call. A loop that makes one call per item turns a small delay into a large one, which is why batching and caching come before micro-optimising code. Pass a deadline down every call chain, so an inner call never outlives the request that needs its answer.
Fallacy 4: the network is secure
NIST's zero trust architecture (SP 800-207, 2020) states the modern version: no implicit trust is granted to assets or user accounts based solely on their physical or network location. Being "inside the network" is not an authentication.
Fallacies 3, 5, 6, 7 and 8 together
Payload sizes, instance counts and addresses all change over time, and the path between two services crosses teams that do not share a release calendar. The habits are the cheap ones: paginate, resolve names at call time, version every contract, and measure what traffic costs. Fallacy 8 is why wire formats are agreed standards and not "whatever our language serialises".
Minimal example: the timeout that tells you nothing
A caller sends a request, waits, and the timer expires. Three different worlds produce exactly this observation.
Figure 1. A timeout hides three outcomes. How to read this: the charge happened, but checkout only saw silence. A request that never arrived, a request that failed, and a reply that never came back all look identical to the caller. Retrying a non-idempotent call here would charge twice.
The fix is not a better timer. It is to make the call safe to repeat: the caller generates a unique key for one logical operation and sends it with every attempt, and the receiver stores the result against the key. The payments lesson shows ShopStream's version.
What to do when a call does not answer
Figure 2. The decision after a timeout. How to read this: the first question is always about repeating, not about waiting longer. The Amazon Builders' Library states the rule: APIs with side effects are not safe to retry unless they provide idempotency.
The same source gives the rest of the policy. Set a timeout on every remote call, choose it from an acceptable false-timeout rate and the matching percentile of measured latency (its example is 0.1 percent and p99.9), and add jitter so retries do not collide again.
In practice: ShopStream's checkout calls a payment provider
This is the realistic example. It is an educational scenario on the course's running example, ShopStream; nothing in it describes a real company, and every figure is illustrative.
This is a realistic scenario, not a case history.
The situation
At the course's illustrative peak of about 58 checkouts per second, ShopStream's checkout calls an external payment provider for each order. During a provider slowdown, shoppers see spinners, press the button again, and the support queue fills with double charges.
Starting architecture
Figure 3. The path the fallacies attack. How to read this: every arrow is a network call that can be slow, lost or repeated. Each hop is also a place where a retry can be added.
Options considered
| Option | Gain | Cost | Risk |
|---|---|---|---|
| A. Long timeout, no retry | Simple | Threads wait for the provider | A slow provider exhausts the checkout service |
| B. Short timeout, retry without a key | Fast recovery | Looks safe | Double charges when the first attempt succeeded |
| C. Short timeout, one retry layer, idempotency key | Safe to repeat, bounded wait | A key store and a retry budget | Needs a clear "payment pending" state |
The decision
Option C. Checkout generates one key per order and passes it down. Only the payments service retries, with a small budget and jitter. Every layer above it has a timeout but never retries. A timeout reaches the shopper as "payment is being confirmed", never as "failed".
Rollout and verification
A rollout would first add the key and log duplicate-key hits, then enable the retry. Verification is a test where the provider acknowledges and drops the reply, asserting one charge and one order. The numbers to watch are the retry ratio, the share of calls that time out, and double-charge tickets.
The 2 a.m. failure
The page reads: checkout threads saturated, provider calls at several times normal volume. The provider slowed down, and the gateway, the checkout service and the payments client each retried, so every shopper click became many provider calls. The Amazon Builders' Library gives this arithmetic: in its example of five layers with three retries each, the failing database sees 243 times the load (243 is 3 to the fifth power, so the article counts three tries per layer).
Figure 4. The failure example: retry amplification. How to read this: each layer multiplies the calls it receives by its own retries. Three layers of three is not nine extra calls, it is a multiple of the original traffic arriving at the one component that is already struggling.
Mitigation: switch off retries in the gateway and checkout service, shed load, and show the pending page. Detection: the ratio of retries to first attempts at every layer. The lasting fix is one retry layer with a token-bucket budget, plus a circuit breaker (resilience patterns).
Architect's notes
- Draw the network on every diagram. If an arrow can fail, say what the caller does when it does.
- Retry in one place. More than one retrying layer multiplies load.
- Treat a timeout as "unknown". Reconcile, do not guess.
- Give every contract an owner and a version.
Common mistakes
- Retrying a non-idempotent call.
- Setting no timeout, or one longer than the caller's own deadline.
- Retrying at every layer.
- Assuming "internal" means trusted.
- Hard-coding addresses or instance counts.
Key takeaways
- A remote call has outcomes a local call does not: lost request, lost reply, slow and failed look alike.
- The eight fallacies are reliability, latency, bandwidth, security, topology, administration, transport cost and homogeneity.
- Make operations safe to repeat before adding retries.
- Retries multiply across layers; budget them in one place.
- Design for partial failure at the interface, not inside each component.
Practice exercises
Exercise 1 (20 min). For each fallacy, name one bug in a system you know that comes from assuming it.
Exercise 2 (30 min). Take one remote call in a project you know. Write its timeout, what a timeout leads to, and whether repeating it is safe.
Exercise 3 (45 min). Compute the load multiplier for three layers that each try a call 4 times in total when the bottom one keeps failing.
Related reading
- Resilience Patterns: timeouts, retries, breakers and bulkheads in depth.
- CAP Theorem and PACELC: what a partition forces.
- ShopStream payments and the ledger: idempotency keys in context.
- ShopStream resilience: timeouts and retry budgets for the whole system.
Official documentation
- Deutsch and colleagues, The Eight Fallacies of Distributed Computing: the list itself.
- Waldo, Wyant, Wollrath and Kendall, A Note on Distributed Computing (Sun Microsystems Laboratories, 1994): partial failure and interface design.
- Bailis and Kingsbury, The Network Is Reliable: a survey of reported network failures.
- Amazon Builders' Library, Timeouts, retries and backoff with jitter.
- NIST, SP 800-207, Zero Trust Architecture (2020).