Backpressure: Bounded Queues, Load Shedding and Credits

What you'll learn: By the end of this lesson you will be able to explain why a queue hides a speed mismatch, name the four things a system can do when a producer outruns its consumer, state what the Reactive Streams request(n) demand rules guarantee, choose between blocking, rejecting and dropping for a kind of work, and run a Java lab that shows each choice's effect on queue size, waiting time and lost work.

A kitchen has a rail where waiters hang order tickets. On a busy night they hang tickets at twice the rate the cook can clear them. Every ticket waits longer than the one before, and by the time one reaches the cook the guest has left. The rail did not solve the speed mismatch; it hid it, and the hidden part grows.

Backpressure is the set of ways a system answers back: "slow down", "not now", or "send me this many". This lesson is the dedicated treatment of the idea. The message queues lesson and ShopStream Stage 5 give it a short section each; here you get the mechanisms, the demand rules and a lab.

Verified with Java 25 (the lab) and the Reactive Streams specification 1.0.4, on 2026-11-08.

The core idea

A producer's rate must end up bounded by its consumer's rate, and something has to say so. When the consumer is slower for a while, the surplus has to go somewhere. There are only four places:

1. A buffer, which holds it and grows, until memory or patience runs out.
2. The bin: refuse or drop some of it (load shedding).
3. The producer: make it wait or stop.
4. Nowhere, because it was never made: the consumer tells the producer how much it can take, and the producer sends no more.

Three examples run through the lesson: a minimal example (the Java lab), a realistic example (ShopStream's courier location pings, in "In practice") and a failure example (a blocking put on a request thread).

Why a queue is not the answer

A queue separates the moment of arrival from the moment of work, which helps with bursts. It does not change the long-run arithmetic: at 100 arrivals a second and 50 completions a second, the queue grows by 50 a second for as long as that lasts. A message behind N others waits about N times the consumer's time per message. An unbounded queue turns overload into a delay that keeps growing, then into an out-of-memory failure.

Figure 1. The mismatch the queue hides. How to read this: the rates are the lab's slow period (illustrative). Nothing in this picture tells the producer to slow down, so the only things that can change are the depth of the queue and the age of its oldest message.

The Reactive Streams specification calls backpressure "an integral part of this model in order to allow the queues which mediate between threads to be bounded." Bounding the queue is the first decision; it forces the second: what happens to the message that does not fit?

Four answers to "the queue is full"

Bounded queues and load shedding

Java's BlockingQueue makes the choice visible in its method names. When the queue is full, add throws IllegalStateException, offer(e) returns false, put(e) blocks until there is space, and offer(e, timeout, unit) waits up to the timeout and then returns false.

Load shedding is refusing work on purpose so the rest can succeed. Done well it is early, explicit (HTTP 503 Service Unavailable, which RFC 9110 describes as the server being unable to handle the request because of temporary overload or maintenance, with a Retry-After header giving a number of seconds or a date) and selective. Google's SRE book describes serving degraded results as one option, and a criticality scheme in which an overloaded task rejects lower-criticality requests sooner. It limits retries too: only at the layer directly above the one that rejected, up to three attempts, and only while retries stay under about 10 percent of a client's requests. Rate limiting is the per-caller version of the same idea.

Making the producer wait

put is the simplest backpressure there is: the producer's thread sleeps until space appears, so the pressure travels up the call stack. It suits a loop that can afford to wait, such as a batch job reading a file. It is dangerous on a request thread, because every waiting thread is a request not being answered (see the failure example).

Credits: the consumer says how much

In credit-based flow control the consumer grants permission to send and the producer sends only against granted credit. Three real systems work this way:

• Reactive Streams and its Java form java.util.concurrent.Flow (since Java 9): Subscription.request(n) "adds the given number n of items to the current unfulfilled demand", and the publisher sends no more than the demand.
• HTTP/2 (RFC 9113): flow control uses WINDOW_UPDATE frames, the initial window is 65,535 octets per stream and for the connection, a sender must respect the receiver's limits, and only DATA frames are subject to it. It covers a single hop, not the whole end-to-end path. The gRPC guide adds that a stream write returning does not mean the data has left the machine: it has been handed to a buffer.
• Kafka consumers pull. Its design documentation says a push system's broker sets the rate, so a consumer "tends to be overwhelmed when its rate of consumption falls below the rate of production", while a pull consumer "simply falls behind and catches up when it can". The backlog stays in the log.

Credits move the buffer to wherever the source can hold it. They work only if the source can slow down: a cursor, a file or a log can wait, but a crowd of shoppers pressing "pay" cannot, and for them the choice is again buffering or shedding.

Figure 2. A decision for each queue. How to read this: start at the top for each queue, not for the system. The two questions that matter are whether the source can wait, and whether the work goes stale. Every path ends with a bound.

Reactive Streams: the rules behind request(n)

java.util.concurrent.Flow holds four interfaces: Publisher (one method, subscribe), Subscriber (onSubscribe, onNext, onError, onComplete), Subscription (request(long) and cancel()) and Processor. The specification, version 1.0.4 at the time of writing, adds rules. These are the ones that define demand:

The specification asks for non-blocking behaviour throughout, so credits must not park threads. Demand is the number of items requested and not yet delivered. A common pattern requests a window up front and one more per finished item, which is what the lab does.

Figure 3. Demand is the only licence to send. How to read this: after the third item the publisher stops, although a fourth is ready. Only the subscriber's next request lets it continue, so the subscriber's queue cannot hold more than its window.

Build it

The lab is examples/backend/labs/backpressure. It simulates a producer, a queue and one consumer on a clock that only moves when told to, so every number is repeatable. Five strategies share one scenario: 600 messages, one every 10 ms, and a consumer that needs 5 ms per message except from t=2000 to t=5999, when a slow downstream makes it 20 ms. PacedPublisher is a small single-threaded Flow.Publisher that follows the demand rules. Its heart is request and the drain loop:

private void request(long n) {
    if (cancelled || terminated) {
        return; // rule 3.6: a request after cancel is a no-op (and nothing follows a terminal signal)
    }
    if (n <= 0) {
        terminated = true; // rule 3.9: signal onError with IllegalArgumentException, then nothing more (1.7)
        subscriber.onError(new IllegalArgumentException("request(n) needs n > 0, got " + n));
        return;
    }
    demand = demand + n < 0 ? Long.MAX_VALUE : demand + n; // rule 3.17: saturate instead of overflowing
    drain();
}

private void drain() {
    if (draining) {
        return; // a request made from inside onNext only adds demand; the running loop picks it up (rule 3.3)
    }
    draining = true;
    try {
        while (!cancelled && !terminated && emitted < total && demand > 0 && clock.nowMillis() >= readyAt) {
            if (demand != Long.MAX_VALUE) {
                demand--;
            }
            int item = emitted++;
            readyAt = clock.nowMillis() + intervalMillis;
            subscriber.onNext(item);
        }
        if (!cancelled && !terminated && emitted == total) {
            terminated = true;
            subscriber.onComplete();
        }
    } finally {
        draining = false;
    }
}

Steps.

1. Run the tests: every message ends in exactly one outcome, bounded queues stay within their bound, stale-dropping keeps processed work under the deadline, a smaller window gives a shorter wait, nothing is shed when the consumer keeps up, and the demand rules hold.
2. Run the demo and compare it with the output below.

Expected output, captured from a real run (Java 25.0.3, 2026-11-08):

Backpressure lab. All times are simulated milliseconds; nothing here reads a real clock.
A producer offers 600 messages, one every 10 ms. A consumer handles one message at a time.
Limits: queue capacity 50, deadline 500 ms, credit window 10.

Part 1. Slow consumer: 5 ms per message, but 20 ms per message from t=2000 to t=5999 (offered load 100/s, slow capacity 50/s).
strategy          offered completed rejected expired maxDepth maxQueueWait  producerWait  finished
--------------------------------------------------------------------------------------------------------
unbounded queue       600       600        0       0      200      2000 ms          0 ms   7000 ms
bounded, reject       600       450      150       0       50       990 ms          0 ms   6250 ms
drop if too old       600       450        0     150       52       500 ms          0 ms   6250 ms
bounded, block        600       600        0       0       50       999 ms       1501 ms   7496 ms
credit (Flow)         600       600        0       0       10       180 ms       1910 ms   7905 ms

Part 2. Healthy consumer: 5 ms per message all the time (capacity 200/s). The limits should never bite.
strategy          offered completed rejected expired maxDepth maxQueueWait  producerWait  finished
--------------------------------------------------------------------------------------------------------
unbounded queue       600       600        0       0        1         0 ms          0 ms   5995 ms
bounded, reject       600       600        0       0        1         0 ms          0 ms   5995 ms
drop if too old       600       600        0       0        1         0 ms          0 ms   5995 ms
bounded, block        600       600        0       0        1         0 ms          0 ms   5995 ms
credit (Flow)         600       600        0       0        1         0 ms          0 ms   5995 ms

Part 3. The Reactive Streams rules the publisher follows (a paced source of 3 items, one ready every 10 ms).
 a) Items are sent only against demand:
  t=0   onSubscribe (no demand yet)
  t=20  tick: item 0 is ready, but demand is 0, so nothing is sent
  t=20  onNext(0)
  t=20  request(2) done, demand left 1
  t=30  onNext(1)
  t=30  tick: demand left 0
  t=40  tick: item 2 is ready, demand is 0, so nothing is sent
  t=40  onNext(2)
  t=40  onComplete
  t=40  request(Long.MAX_VALUE) done, emitted 3 of 3
  t=40  request(1) after the terminal signal: nothing happens
 b) request(0) is an error:
  t=0   onSubscribe (no demand yet)
  t=0   onError(IllegalArgumentException: request(n) needs n > 0, got 0)
  t=0   request(5) after onError: nothing happens, emitted 0
 c) After cancel nothing more is sent:
  t=0   onSubscribe (no demand yet)
  t=0   onNext(0)
  t=10  cancelled, then request(5) and a tick: emitted 1 of 3

Part 2 is the control: with a healthy consumer every strategy gives the same result, so backpressure costs nothing without overload. Part 3 traces the rules: no onNext without demand, request(0) ends the stream with an error, a cancelled stream stays quiet.

Break it. In Simulation.step, replace the rejecting condition with false: the reject row becomes the unbounded row and its test fails. Delete demand > 0 && from the drain loop: the publisher ignores demand and several tests fail. (Both changes together failed 6 of the 22 tests; the files were then restored.)

Stretch. Shed by priority, or adapt the credit window to the age of the oldest message.

What the model leaves out. Real threads, networks, retries and priorities. PacedPublisher is a teaching model of the rules a single thread can follow, not a complete or thread-safe implementation.

The other side of this topic

On the front end the slow consumer is often the screen: a feed of updates can arrive faster than the interface can paint it. The same choices apply: drop or merge updates, slow the source, or let the backlog grow. The Frontend System Design track plans a lesson on this; it is not published yet, so there is no link.

In practice: courier pings against the tracking store

This is the realistic example. It is an educational scenario on the course's running example, ShopStream, and nothing in it describes a real company. Every figure is illustrative.

This is a realistic scenario, not a case history.

The situation

Couriers' phones send location pings to an ingest service so shoppers can watch a map. Assume 2,000 active couriers pinging every 5 seconds: 400 pings a second. The tracking service writes them to a time-series store (data model) that takes about 600 writes a second, but only about 250 during a background compaction. For ten minutes the ingest service receives 150 more pings a second than the store accepts.

Starting architecture

Figure 4. The design under discussion. How to read this: the red box is the unbounded buffer. During compaction it grows by 150 pings a second, about 90,000 pings in ten minutes, and the map shows positions that get older by the minute.

Options considered

The decision

Option D for pings: a position is worth only until the next one arrives, so the writer takes the latest entry for each courier. The buffer never holds more than 2,000 entries, whatever the store does. Payments are different: they must not be dropped and do not go stale, so their queue stays bounded and rejects early with 503 and Retry-After, as in Stage 5. This follows Figure 2.

Rollout and verification

Run the new buffer beside the old path. Watch the age of the oldest unwritten entry (alert at 30 seconds, illustrative), pings superseded per minute, store write latency and ingest threads in use. Rehearse the compaction in a test environment by slowing the store, and confirm the buffer stays at or below 2,000 and no request thread waits.

The 2 a.m. failure

The page reads: ingest health checks failing. The team had briefly shipped option B. The store slowed, the bounded queue filled, and every request thread blocked inside put. With no free thread the service stopped answering even its health check, the orchestrator restarted it, and thousands of phones reconnected at once. A thread dump shows many threads parked in put. Mitigate with offer and a short timeout, answering 503 with Retry-After; fix with option D and a thread budget separate from the queue (see resilience patterns).

Architect's notes

• Bound every queue, then decide what the bound does. An unbounded queue is a decision not to decide.
• Match the answer to the work. Positions and prices can be dropped or merged; payments and orders are refused early with a clear error.
• Never block a request thread on a full queue. Use offer with a timeout, or credits.
• Measure age, not only depth. A short queue with an old head is worse than a long one that drains.

Common mistakes

• An unbounded queue "for safety". It turns overload into delay, then a crash.
• put on a thread that also serves requests.
• Silent drops. Refuse loudly; never drop acknowledged work.
• Requesting Long.MAX_VALUE out of habit. It switches the bound off.
• Treating cancel as instant. Items can still arrive after it.

Key takeaways

• A queue hides a speed mismatch; backpressure answers it.
• The surplus goes to a buffer, the bin, the producer, or is never made.
• Bound every queue, and choose reject, wait, drop or credits by what the work is.
• Reactive Streams demand is cumulative: never more items than requested, request(0) is an error, Long.MAX_VALUE means unbounded.
• Credit-based systems (Reactive Streams, HTTP/2, Kafka pull) put the buffer where the source can hold it.
• Measure the age of the oldest item and the amount refused.

Practice exercises

Exercise 1 (30 min). From part 1, list each strategy's completed share and longest wait. Which suits payments, and which location pings?

Exercise 2 (45 min). Run the lab with a credit window of 40, 20, 10 and 5. Record the longest queue wait and the finish time, and explain the trend.

Exercise 3 (1 hour). Do "Break it" and say which promise each failing test protects.

Exercise 4 (1 hour). Implement the "latest per courier" buffer and test that it never holds more entries than there are couriers.

Related reading

• Message Queues and Asynchronous Work: acknowledgements, visibility and ordering.
• Resilience Patterns: bulkheads, timeouts and load shedding in the wider picture.
• Rate Limiting: per-caller limits at the entry point.
• ShopStream Stage 5: backpressure on the payment queue.

Official documentation

• Reactive Streams specification 1.0.4: the demand rules.
• java.util.concurrent.Flow (JDK 25 source with its documentation).
• java.util.concurrent.BlockingQueue: add, offer, put on a full queue.
• RFC 9113, HTTP/2: section 5.2, flow control.
• gRPC flow control guide.
• Google SRE book, Handling Overload.
• Apache Kafka design: push vs pull.