What you'll learn: By the end of this lesson you will be able to define latency, throughput and concurrency without mixing them up, use Little's law to work out how many requests a service holds in flight, explain with a small simulation why latency climbs sharply as a server nears its capacity, compute percentiles and say what an average hides, name the techniques that trade one for the other, and recognise the retry spiral that turns a slow dependency into an outage.
A coffee shop has one barista and a line. "How long did my coffee take?" is latency. "How many coffees leave the shop each hour?" is throughput. A second barista raises the hourly count without speeding up any single coffee, and a long line slows everyone even when the baristas are quick. Design reviews go wrong when someone says "make it faster" without saying which of the two they mean.
This lesson gives the definitions, one law that ties the numbers together, one small queueing model and the arithmetic of percentiles. It assumes the estimation lesson. Dedicated lessons on queueing and tail latency are planned and not published yet.
The Java program ran as a single-file program on JDK 26.0.1 on 2026-11-08 and uses nothing newer than Java 17. Numbers marked "illustrative" are assumptions, not measurements.
The core idea
Latency is how long one request takes. Throughput is how many requests finish per unit of time. Concurrency is how many requests are in the system at once. Little's law ties the three together, and all three behave in a particular way when a service nears its limit.
The Google SRE book defines latency as "the time it takes to service a request" and says to keep successful and failed requests apart: a fast error makes an average look better than the user's experience.
| Term | Meaning here |
|---|---|
| Service time | Time a worker spends on one request |
| Wait time | Time queued before a worker starts |
| Latency (response time) | Wait plus service, as the caller sees it. Some teams use "latency" for network delay only, so say which |
| Throughput | Requests completed per second |
| Concurrency | Requests in the system now, waiting or served |
| Utilization | Share of time a resource is busy, 0 to 1 |
Three examples run through the lesson: a minimal example (one worker), a realistic example (ShopStream's browse path at peak) and a failure example (the retry spiral).
A minimal example: one worker
One worker, 50 ms of work per request. With nothing waiting, latency is 50 ms. The worker finishes one request every 0.05 s, so throughput tops out at 1 / 0.05 = 20 requests per second. A second worker lifts capacity to 40 per second, but each request still takes 50 ms. Workers raise capacity and leave latency alone. To cut latency you shorten the work itself (a faster query, a cache hit, fewer steps) or run parts in parallel.
Figure 1. A line in front of workers. How to read this: latency is the time in the line plus the time at a worker. Throughput is the rate at which requests leave on the right. More workers widen the right side; they do nothing for the work each request needs.
Little's law: how many requests are in flight
John Little proved in 1961 that the long-run average number of items in a queueing system, L, equals the average arrival rate, λ, times the average time an item spends in the system, W:
L = λ × W
His 2011 review notes that the result needs no assumption about the arrival pattern, the service-time distribution or the order of service, and that it applies to a single queue or a network of them. "In the system" includes both waiting and being served. Given two of the three numbers you can compute the third; with a fixed arrival rate, the only way to hold fewer requests in flight is to make each spend less time there. For a service that drops no work and is in a steady state, the arrival rate equals throughput, so: requests in flight = throughput × average latency.
Realistic example. The ShopStream brief assumes about 2,300 browsing requests per second at peak (an illustrative assumption). Say the catalogue call averages 80 ms (illustrative).
| Average latency | Requests in flight, 2,300 × W |
|---|---|
| 80 ms | 184 |
| 160 ms | 368 |
| 400 ms | 920 |
Add a fixed pool of 200 worker threads (illustrative). Rearranged, the law says a pool of L threads serves at most L / W requests per second. At 80 ms that is 200 / 0.08 = 2,500 per second, just above the 2,300 demand. If latency doubles to 160 ms, the same pool serves at most 1,250 per second, and 1,050 requests per second queue. No thread died; a dependency got slower and the pool's capacity halved.
The law is about long-run averages. It says nothing about the slowest requests, and it tells you what the numbers must be, not how to improve them.
Why latency climbs as a server fills up
Requests do not arrive evenly: sometimes two arrive together and the second waits. The busier the server, the less idle time it has to catch up.
The simplest model is the M/M/1 queue. MIT's manufacturing-systems course notes state its assumptions: one server with an exponential service rate μ, an unlimited waiting area and an arrival rate λ below μ. With utilization ρ = λ / μ, the average number in the system is ρ / (1 − ρ), and Little's law then gives the average time in the system:
W = 1 / (μ − λ)
For a 10 ms mean job, μ = 100 per second. At 50 percent utilization the formula gives 20 ms, at 90 percent 100 ms, at 99 percent 1,000 ms. The curve is gentle, then nearly vertical.
Figure 2. One request, split into wait and service. How to read this: the service time does not change as load rises. Only the wait grows, and the wait is the part that explodes near full utilization.
The formula gives averages. This simulation of the same queue shows the spread: random arrivals and job lengths, a fixed seed so every run prints the same output. A request starts when it arrives or when the previous one finishes, whichever is later.
import java.util.Arrays;
import java.util.Random;
public class QueueDemo {
public static void main(String[] args) {
double mu = 100.0; // one server, 100 requests/s (mean service time 10 ms)
int n = 2_000_000;
System.out.println("util mean ms p50 ms p95 ms p99 ms (formula: mean, p99)");
for (double rho : new double[] {0.5, 0.8, 0.9, 0.95}) {
double lambda = rho * mu;
Random rng = new Random(42); // fixed seed: the same output on every run
double[] response = new double[n];
double arrive = 0, previousDeparture = 0, sum = 0;
for (int i = 0; i < n; i++) {
arrive += -StrictMath.log(1 - rng.nextDouble()) / lambda; // exponential gaps
double service = -StrictMath.log(1 - rng.nextDouble()) / mu; // exponential work
double start = Math.max(arrive, previousDeparture); // wait if the server is busy
previousDeparture = start + service;
response[i] = previousDeparture - arrive; // wait + service
sum += response[i];
}
Arrays.sort(response);
System.out.printf("%.2f %8.1f %6.1f %6.1f %6.1f (%.1f, %.1f)%n", rho, 1000 * sum / n,
1000 * response[(int) Math.ceil(0.50 * n) - 1], 1000 * response[(int) Math.ceil(0.95 * n) - 1],
1000 * response[(int) Math.ceil(0.99 * n) - 1], 1000 / (mu - lambda),
1000 * StrictMath.log(100) / (mu - lambda));
}
}
}Save it as QueueDemo.java and run java QueueDemo.java. Output from a real run (JDK 26.0.1, 2026-11-08, identical on a second run):
util mean ms p50 ms p95 ms p99 ms (formula: mean, p99)
0.50 20.1 13.9 60.0 92.1 (20.0, 92.1)
0.80 50.4 35.0 150.9 231.2 (50.0, 230.3)
0.90 101.5 70.8 303.1 457.1 (100.0, 460.5)
0.95 202.3 142.8 599.6 888.6 (200.0, 921.0)The job always takes 10 ms on average. At 50 percent utilization the mean latency is twice that; at 95 percent it is twenty times, and the p99 is nearly 900 ms. The simulated mean is within about 1.5 percent of 1 / (μ − λ). The p99 formula, ln(100) / (μ − λ), holds for this queue with first-come-first-served service; the simulation is close, furthest off at 95 percent (about 3.5 percent low), where a nearly full queue converges slowly. Treat the model as a shape, not a prediction: real services have several servers, uneven jobs and retries, and Little's 2011 paper notes that large queues can slow the service rate itself, so latency can rise even faster.
Percentiles: what an average hides
A percentile is a position in the sorted list of timings: p99 is a latency that 99 percent of requests are at or below. There are several ways to compute it; this lesson uses nearest rank: sort n values and take the one at position ceil(p / 100 × n).
Worked example. One thousand requests: 980 take 20 ms and 20 take 2,000 ms.
| Statistic | Value | How |
|---|---|---|
| Mean | 59.6 ms | (980 × 20 + 20 × 2,000) / 1,000 |
| p50, p95 | 20 ms | positions 500 and 950 |
| p99 | 2,000 ms | position 990 |
The mean is three times the typical request and matches nobody's experience. Make exactly 1 percent slow (990 at 20 ms, 10 at 2,000 ms) and the mean is 39.8 ms, p99 is 20 ms and p99.9 is 2,000 ms. Where the slow requests start decides which percentile shows them, so state the percentile you promise and how you compute it.
The SRE book makes the same point with a service averaging 100 ms at 1,000 requests per second where 1 percent of requests might take 5 seconds, and notes that one backend's 99th percentile can easily become the median of the front end calling it. The arithmetic: if a page makes 100 backend calls and each has a 1 percent chance of being slower than its p99, the chance that at least one is slow is 1 − 0.99^100 = 63.4 percent (9.6 percent for 10 calls). Fan-out turns a rare event into a common one.
Two rules follow. Do not average percentiles across servers; collect counts in latency buckets, as the SRE book recommends, because buckets can be merged. And measure failed requests separately.
Figure 3. A first diagnostic. How to read this: ask the questions in order. They separate "each request does too much work" from "no spare capacity" from "most requests are fine and a few are not".
Trading one for the other
| Technique | Throughput | Latency | The cost |
|---|---|---|---|
| More workers or replicas | Rises until a shared resource is the limit | Per request unchanged; less waiting under load | Money; the database becomes the bottleneck |
| Batching | Rises sharply | Each item waits for the batch to fill and run | Partial-failure handling |
| Cache in front of the work | Rises (less work) | Falls for hits | Stale data (see caching basics) |
| Queue between caller and worker | Absorbs bursts | Caller returns early; the work waits in line | A line that can grow without limit |
| Load shedding, concurrency limit | Capped at what can finish | Accepted requests stay fast | Some callers get an error |
| Waiting for replicas to confirm | Falls | Rises | Stronger consistency (see CAP and PACELC) |
An illustrative batching case: a single-row insert takes 5 ms, so one connection does 200 rows per second. A 100-row batch takes 30 ms, about 3,333 rows per second on the same connection, but a row waits for the batch to fill plus 30 ms. Throughput is over sixteen times higher and each row's latency is worse. Whether that is right depends on whether anyone waits for that row.
Failure example: the retry spiral
Figure 4. How slowness becomes an outage. How to read this: the red dashed arrow is the harmful path. Threads wait on the slow call, callers time out and retry, and the retries add load to a service that is already full.
- A dependency's latency rises. Throughput is unchanged and W grew, so Little's law says requests in flight rise in proportion.
- The fixed pool fills. New requests queue, so their latency now includes waiting.
- Callers time out and retry. Demand rises just as capacity fell.
- Demand passes capacity, the queue grows, and even requests that never touch the slow dependency wait.
What stops it: a timeout on every outgoing call, a retry budget with backoff and random delay, a concurrency limit per dependency (a bulkhead) so one slow call cannot take every thread, and shedding or degrading (return the page without recommendations). See resilience patterns and rate limiting.
In practice: a promotion hour on ShopStream's browse path
This is an educational scenario on the course's running example, ShopStream. Every figure is illustrative.
This is a realistic scenario, not a case history.
The situation
A promotion starts at noon. The browse path must handle about 2,300 requests per second, with a p99 under 400 ms on the catalogue call (an illustrative target). A load test shows a mean of 80 ms at that rate and a p99 near 600 ms.
Starting architecture
Figure 5. Four instances with 50 threads each. How to read this: 200 threads at 80 ms finish 2,500 requests per second, so the pool already runs at about 92 percent of capacity at peak, with almost no slack.
Options considered
| Option | Effect | Risk |
|---|---|---|
| A. Raise each pool to 100 threads | Capacity at 80 ms doubles | The database sees twice the concurrent queries; the bottleneck moves |
| B. Cache the hottest catalogue pages | Lower W, so fewer in flight and more capacity | Stale prices; invalidation bugs |
| C. Add instances | More threads | Same database limit as A, plus cost |
| D. Concurrency limit with early rejection | Accepted requests stay fast in overload | Some shoppers see "try again" |
The decision
B and D. Caching removes work, which lowers latency and raises capacity together (lower W at the same λ means fewer requests in flight). A limit just below what the database can take, with a quick and clear rejection, stops overload from turning into queueing. A and C wait for a test that shows database headroom.
Rollout and verification
Load-test to 1.5 times peak (3,450 requests per second) with an artificial slowdown on one dependency. Watch requests in flight and check it against rate times mean latency, plus p50, p95 and p99 from bucketed counts, queue wait and rejections. The targets: p99 under 400 ms at peak and graceful rejection above it. Release to a small share of traffic first.
The 2 a.m. failure
p99 and errors rise while traffic is normal. A recommendations call has gone from about 20 ms to 400 ms, so in-flight requests head toward 920 (2,300 × 0.4) against 200 threads, and retries add load. First checks: the in-flight gauge, per-dependency latency, and a thread dump showing threads parked on that call. Mitigation: serve pages without recommendations, lower that call's timeout, let the limit reject early. Lasting fix: a bulkhead and timeout per dependency, a retry budget, and a load test that includes a slow dependency.
Architect's notes
- Name the percentile, window and rate in every requirement: "p99 under 400 ms over five minutes at 2,300 requests per second".
- Do not plan to run hot. Latency is gentle at low utilization and steep near the limit.
- Decide who is waiting. Batch and queue where nobody waits; keep the synchronous path short where someone does.
- Decide in advance what is rejected or degraded under overload.
Common mistakes
- Reporting the mean, or averaging percentiles across servers.
- Sizing a pool for the average and ignoring a slower dependency.
- Retrying without a timeout, budget or backoff, and treating an unbounded queue as free.
- Counting fast errors as good latency, or saying "fast" without saying which number.
Key takeaways
- Latency is time per request, throughput is requests per unit time, concurrency is requests in the system.
- L = λW holds for long-run averages in a stable system; use it to size pools and queues.
- With fixed workers, throughput is at most workers divided by latency, so a slower dependency lowers capacity.
- Latency is shallow at low utilization and rises steeply near the limit.
- A percentile needs a stated method; 100 calls at p99 each leave a 63.4 percent chance of at least one slow call.
- Timeouts, retry budgets, bulkheads and shedding keep a slow dependency from becoming an outage.
Practice exercises
Exercise 1 (20 min). A service has 64 threads and a request averages 40 ms. What is its highest sustainable throughput, and what is it at 100 ms?
Exercise 2 (30 min). At 1,200 requests per second and 250 ms mean latency, how many requests are in flight?
Exercise 3 (45 min). Run QueueDemo.java with rho set to 0.99. What happens to the mean and p99, and why does the gap to the formula change?
Exercise 4 (30 min). Compute p50, p95 and p99 by nearest rank for 2,000 requests, 1,950 at 30 ms and 50 at 900 ms. Repeat with 20 slow ones.
Exercise 5 (1 hour). Pick a request path you know. Write its latency and throughput requirement, estimate requests in flight and name the first limit you would reach.
Related reading
- Observability and SLOs: percentiles and SLIs.
- Resilience patterns: timeouts, retries, bulkheads.
- ShopStream Stage 2: the browse path with caches.
Official documentation
- Little, Little's Law as Viewed on Its 50th Anniversary (Operations Research, 2011).
- MIT OpenCourseWare 2.854, M/M/1 Queue.
- Google, Site Reliability Engineering: Monitoring Distributed Systems.
- Dean and Barroso, The Tail at Scale (Communications of the ACM, 2013).