What you'll learn: By the end of this lesson you will be able to say what a
ThreadPoolExecutordoes with each submitted task (start a thread, queue it or reject it), explain why an unbounded queue makes the maximum pool size irrelevant, size a pool from arrival rate and time per request with Little's law, compose and failCompletableFuturestages without losing errors, recognise thread starvation when a pool waits on itself or on blocking calls, and use a Java lab that pins these rules with counters and tests instead of timings.
Picture a supermarket with a fixed number of open checkout lanes. A shopper goes straight to a free lane; when every lane is busy, they join one shared queue; if the queue gets long, the manager opens extra lanes, up to the tills the shop owns; if the queue is full and every till is open, the shopper is turned away at the door. A Java thread pool is that picture: lanes are threads, and "turned away" is a rejection. The owner must answer how many lanes, how long a queue, and what happens at the door.
The previous lesson showed how threads share memory. This one is about running many tasks on few threads and chaining their results without blocking.
Verified with the Java SE 25 API documentation for ThreadPoolExecutor, Executors, ExecutorService, CompletableFuture and ForkJoinPool, on 2026-11-19. Lab run with Java 25.
The core idea
A thread pool reuses a bounded set of threads for a stream of tasks, and the real design decision is the queue and the rejection rule in front of those threads, not the thread count alone. The Javadoc of ThreadPoolExecutor names two purposes: better performance "due to reduced per-task invocation overhead", and "a means of bounding and managing the resources, including threads, consumed when executing a collection of tasks". CompletableFuture then lets you describe what happens after a task finishes, as stages, without a thread waiting for it.
Three illustrations run through the lesson, each labelled:
- Minimal example: a burst of seven blocking tasks and what four different pool set-ups do with it, in the Java lab.
- Realistic example: ShopStream's product page fanning out to a price service and a stock service, in "In practice".
- Failure example: a slow downstream service that fills an unbounded queue, and a pool whose threads all wait on each other (starvation).
What the pool does with a task
ThreadPoolExecutor has two sizes, corePoolSize and maximumPoolSize, plus a work queue and a rejection handler. The Javadoc gives the order of decisions for each call to execute: "if fewer than corePoolSize threads are running, a new thread is created to handle the request, even if other worker threads are idle. Else if fewer than maximumPoolSize threads are running, a new thread will be created to handle the request only if the queue is full." In between, "if corePoolSize or more threads are running, the Executor always prefers queuing a request rather than adding a new thread". A request that can be neither queued nor given a new thread is rejected.
Figure 1. The order of decisions on every submission. How to read this: follow the arrows from the top. A new thread is the first choice up to the core size, the queue is the second, extra threads are the third and only then does rejection happen. Because the queue is tried before extra threads, a queue that never fills means the maximum is never reached.
The queue decides the behaviour
The Javadoc names three queueing strategies.
| Queue | What the Javadoc says | Effect on the pool |
|---|---|---|
Direct hand-off (SynchronousQueue) |
A queue attempt "will fail if no threads are immediately available to run it, so a new thread will be constructed"; it generally needs an unbounded maximum to avoid rejection | A thread per concurrent task; growth is unbounded if the maximum is |
Unbounded (LinkedBlockingQueue with no capacity) |
"no more than corePoolSize threads will ever be created. (And the value of the maximumPoolSize therefore doesn't have any effect.)"; it "admits the possibility of unbounded work queue growth" | Smooths bursts, but a sustained overload grows the queue without limit |
Bounded (ArrayBlockingQueue) |
Prevents resource exhaustion with finite maximum sizes, but "can be more difficult to tune and control" | Limits waiting work; the maximum and the rejection rule start to matter |
The Executors factory methods hide this choice. newFixedThreadPool uses "a shared unbounded queue", so the thread count is capped and the waiting line is not. newCachedThreadPool creates threads as needed and removes ones unused for sixty seconds. newThreadPerTaskExecutor starts a new thread per task and states that "the number of threads created by the Executor is unbounded". Each decides where overload goes; building a ThreadPoolExecutor directly makes that decision visible in code review.
What happens at the door: rejection policies
When the pool is saturated, the handler decides. The Javadoc lists four predefined policies:
| Policy | Behaviour (Javadoc) | Use it when |
|---|---|---|
AbortPolicy (default) |
Throws a runtime RejectedExecutionException |
The caller can turn the failure into a fast error such as a 503 |
CallerRunsPolicy |
"the thread that invokes execute itself runs the task", a "simple feedback control mechanism that will slow down the rate that new tasks are submitted" | The submitter can afford to do the work and slowing it is the goal |
DiscardPolicy |
The task "is simply dropped"; for the rare cases where completion is never relied on | Almost never for requests |
DiscardOldestPolicy |
Drops the task at the head of the queue and retries; "rarely acceptable" | Almost never |
Rejection is the pool telling the caller that the system is full. Backpressure and load shedding and retry storms cover what the caller does next.
Sizing a pool with Little's law
Little's law says the average number of requests inside a stable system equals the arrival rate times the time each request spends in it. If every in-flight request holds one thread, that is the number of threads kept busy on average: a floor for the average, not a safe maximum, so add headroom for bursts. Run backwards, it gives a ceiling: N threads at S seconds per request complete at most N divided by S requests per second.
The illustrative numbers in the lab (200 requests per second, 50 ms or 400 ms per request, 16 threads) show how fast the answer moves when a downstream call slows down: 10 threads are enough at 50 ms, but 80 are needed at 400 ms, and a pool of 16 can then complete at most 40 requests per second. The Javadoc's comment on bounded queues points the same way: "if tasks frequently block (for example if they are I/O bound), a system may be able to schedule time for more threads than you otherwise allow". The measure to watch is how long tasks wait in the queue compared with how long they run.
From tasks to pipelines: Future and CompletableFuture
ExecutorService.submit "extends base method Executor.execute(Runnable) by creating and returning a Future". The difference matters for failures. The afterExecute documentation says that tasks wrapped by submit are caught and held in the task object, "so they do not cause abrupt termination, and the internal exceptions are not passed to this method". A task submitted with submit that throws is silent until someone calls Future.get. A plain Future can only be waited on; CompletableFuture can also be completed by hand and chained into stages that run when earlier stages finish.
| Need | Method | Rule from the Javadoc |
|---|---|---|
| Transform a result | thenApply |
A new stage with the function's result |
| Chain a call that itself returns a future | thenCompose |
Completes with the value of the stage the function returns; "analogous to Optional.flatMap"; the function "must arrange eventual completion of its result" |
| Combine two independent results | thenCombine |
Runs when "this and the other given stage both complete normally" |
| Wait for several | allOf |
Completes when all complete; "the results, if any, ... are not reflected in the returned CompletableFuture, but may be obtained by inspecting them individually" |
| Take the first | anyOf |
Completes with the same result as the first to complete |
| Put a time limit | orTimeout, completeOnTimeout |
Fails with a TimeoutException, or completes with a fallback value |
Figure 2. Two independent calls and one join. How to read this: the request thread describes the two calls and the join, then does not have to wait on a thread per call. The executor box is the part people forget: every async stage runs on some executor, and which one is a choice.
Which thread runs a stage
The Javadoc says: "All async methods without an explicit Executor argument, as well as those involving delays are performed using the ForkJoinPool.commonPool()", and defaultExecutor() is the common pool "if it supports more than one parallel thread, or else an Executor using one thread per async task". Non-async dependents "may be performed by the thread that completes the current CompletableFuture, or by any other caller of a completion method". So a stage with no executor can run on the common pool, or on whichever thread finished the previous step. Passing an executor to the ...Async methods fixes the place.
Failure, timeouts and cancellation
When a stage fails, dependent stages are skipped and complete exceptionally. join() throws an unchecked CompletionException with the underlying exception as its cause, and get() throws an ExecutionException with the same cause. The handling methods differ in what they do with the outcome:
exceptionallyruns only on failure and supplies a replacement value; later stages then run normally.handleruns on either outcome with the result or the exception and returns a new value.whenCompleteobserves without translating. If the action itself throws, a normal stage completes exceptionally with the action's exception, and a failed stage keeps its own exception.
Two behaviours are easy to get wrong. allOf is not complete until every input is, and if any input failed it also fails, "with a CompletionException holding this exception as its cause". And cancel does not stop work: it "completes this CompletableFuture with a CancellationException", and the parameter mayInterruptIfRunning "has no effect in this implementation because interrupts are not used to control processing". The computation keeps running and its result is discarded; orTimeout likewise fails the future, not the work. Exceptions can arrive wrapped or unwrapped depending on how the stage failed, so unwrap the cause before you test its type.
Starvation: when every thread is waiting
A pool starves when all its threads are blocked on something that only another task in the same pool can provide, or on a slow dependency, so that independent tasks cannot start. Two shapes matter.
Figure 3. A fast task waits behind blocked ones. How to read this: task C needs no downstream call, yet it cannot run, because both threads are held by tasks that do. Nothing is deadlocked; the delay ends when a call returns. If the thing a thread waits for needs a thread of the same pool, it can wait forever.
The first shape is a pool whose threads block on a slow call. The second is a task that waits, inside the pool, for another task that needs the pool; the lab's single-thread test holds the lone thread that way, and an independent task cannot start until the dependency is satisfied from outside. The same happens with the common pool: block all its threads, and supplyAsync without an executor cannot run. The ForkJoinPool Javadoc says that adjustments to keep parallelism "are not guaranteed in the face of blocked I/O or other unmanaged synchronization", and offers managedBlock for blocking the pool is told about. The simpler rule is to put blocking work on its own named executor, so a slow dependency exhausts only its own pool.
Build it: pool rules and starvation as exact assertions
This is the minimal example. Timing makes pool demos flaky, so the lab measures none: every pool thread blocks on one latch, a burst is submitted, and the pool's state is read after the last submission. With every thread busy, the numbers follow the rules above and nothing else.
Goal. See that a queue and a maximum interact (and that an unbounded queue ignores the maximum), see the rejection policies, and pin the CompletableFuture and starvation rules in tests.
Setup. From the repository's examples/backend folder:
./gradlew :labs:executor-sizing:check --console=plain
./gradlew -q :labs:executor-sizing:run
The first command runs 25 tests (13 on CompletableFuture, 8 on pool behaviour, 4 on starvation). The second prints the table below. The code is in examples/backend/labs/executor-sizing/src/main/java/buildingai/labs/executorsizing/.
How it works. PoolProbe.burst builds a pool from the arguments, submits the tasks, and returns the pool size, the queue length, the number of rejections and the number of tasks the caller ran itself:
ThreadPoolExecutor pool = new ThreadPoolExecutor(core, max, 60, TimeUnit.SECONDS, queue,
namedFactory(WORKER_PREFIX),
handler == null ? new ThreadPoolExecutor.AbortPolicy() : handler);
Sizing holds the Little's law arithmetic. The starvation tests block every thread and assert that a victim task has not run after a 300 ms look; while every thread is blocked that is always true, so it cannot flake.
Expected output, captured from a real run (Java 25):
Burst of blocking tasks, every pool thread busy until the burst ends. Pool state after the last submission.
# setup (core, max, queue, rejection) tasks threads in queue rejected ran on caller
----------------------------------------------------------------------------------------------------------
1 core 2, max 4, ArrayBlockingQueue(2), abort 7 4 2 1 0
2 core 2, max 10, unbounded LinkedBlockingQueue 20 2 18 0 0
3 core 0, max 3, SynchronousQueue (handoff) 4 3 0 1 0
4 as row 1 with CallerRunsPolicy 7 4 2 0 1
Little's law, threads busy on average = requests per second x seconds per request (illustrative numbers):
200 requests/s x 50 ms -> 10 threads busy on average
200 requests/s x 400 ms -> 80 threads busy on average
a pool of 16 threads, 50 ms per request -> at most 320 requests/s
a pool of 16 threads, 400 ms per request -> at most 40 requests/s
What the output shows. Row 1 is Figure 1 in numbers: two core threads, two queued, two more threads because the queue was full, and the seventh task rejected. Row 2 has a maximum of 10 and ends with 2 threads and 18 tasks waiting: the queue never fills, so the maximum is never used. Row 3 has no queue: three threads for three concurrent tasks and a rejection for the fourth. Row 4 is row 1 with CallerRunsPolicy: the seventh task runs on the submitting thread instead of failing. The last four lines are arithmetic, not measurements.
Break it. In Demo, change row 1 to a queue of 3 and predict the new row before you run it. Then, in StarvationTest, replace the blocking call in the fixed-pool test with Thread.sleep(1_000): the victim is only delayed, not starved, because a sleep ends by itself. Put the code back and run the tests again.
Stretch. Add a pool subclass that records queue wait per task with beforeExecute; that is a measurement, so it belongs in a benchmark, not a golden file. What the model leaves out: CPU contention, real I/O and how a framework sizes its own pools.
Choosing a pool: the trade-offs
Decide first whether the task computes or waits, because that sets how many threads are useful; then decide what overload should do; and keep mixed kinds of work in separate pools.
| Option | Gain | Cost | Wrong when |
|---|---|---|---|
Fixed pool, unbounded queue (newFixedThreadPool) |
Simple, thread count capped | Overload becomes memory growth and delay | Load can exceed capacity for long |
Bounded pool and queue, AbortPolicy |
Overload is visible and cheap | Callers must handle rejection | Callers cannot degrade |
Bounded, CallerRunsPolicy |
Natural slowing of the submitter | The submitter's thread does the work | The submitter is a latency-critical event loop |
| Cached pool | Reuses idle threads, starts new ones on demand | Thread count follows load | A slow dependency makes threads pile up |
Default CompletableFuture executor |
Nothing to configure | Shares the common pool | Any stage blocks |
| Thread per task on virtual threads | No pool to size for blocking tasks | The downstream limit must come from elsewhere | CPU-bound tasks |
The last row is the topic of a separate lesson on virtual threads.
In practice: ShopStream's product page fan-out
This is the realistic example, an educational scenario on the course's running example ShopStream; nothing in it describes a real company. Numbers other than ShopStream's shared peak (about 2,300 browsing requests per second across the platform) are illustrative.
This is a realistic scenario, not a case history.
The situation
The catalogue service builds a product page from two calls: a price call and a stock call. At peak the platform serves about 2,300 browsing requests per second, spread over many instances; assume, for illustration, one instance sees 200 per second, and that a request takes 50 ms when both dependencies are healthy. By Little's law that is about 10 threads busy on average. The team gave the service one fixed pool of 16 threads with the default unbounded queue, and used CompletableFuture.supplyAsync with no executor for the two calls.
Starting architecture
Figure 4. Two pools, no limits between them. How to read this: a request thread builds the stages and waits for the join; the stages run on the common pool, which also runs everything else in the process that uses a default executor. Neither pool has a bound on its waiting line that the service can see.
Options considered
| Option | Behaviour | Risk |
|---|---|---|
| A. Keep it, add more threads | A larger fixed pool | Moves the limit; the queue is still unbounded |
B. Bounded pools per dependency, explicit executor for each, timeouts, AbortPolicy mapped to a fast error or a fallback |
Overload is visible; one slow dependency cannot take all threads | More configuration to size and alert on |
| C. Virtual threads | No per-task pool to size for the blocking calls | The downstream still needs its own limit; the pattern is covered in its own lesson |
The decision
Option B for this release. The stock call gets its own bounded executor (with a bounded queue) and completeOnTimeout with a value that means "stock unknown", so a slow stock service degrades the page and does not fail it. The price call gets its own executor and orTimeout, because a page with no price is not worth serving. A rejection from either becomes a fast error response. The sketch below is not part of the lab and was not run:
// Sketch, not run: placeholders such as Price, Stock and ProductPage stand for the service's own types.
ExecutorService stockPool = new ThreadPoolExecutor(8, 8, 0, TimeUnit.SECONDS,
new ArrayBlockingQueue<>(32), new ThreadPoolExecutor.AbortPolicy());
CompletableFuture<Stock> stock = CompletableFuture
.supplyAsync(() -> stocks.lookup(id), stockPool)
.completeOnTimeout(Stock.UNKNOWN, 300, TimeUnit.MILLISECONDS);
CompletableFuture<Price> price = CompletableFuture
.supplyAsync(() -> prices.lookup(id), pricePool)
.orTimeout(300, TimeUnit.MILLISECONDS);
CompletableFuture<ProductPage> page = price.thenCombine(stock, ProductPage::new);
Rollout and verification
The team ships it behind a flag to one instance and watches, per pool: active threads, queue length, rejections and queue wait compared with run time. The lab's burst table is the model for a test: submit more blocked tasks than threads plus queue and assert the rejection count, with latches and no sleeps. The fallback is tested by completing the stock future by hand, not by waiting for a real timeout.
The 2 a.m. failure
The page reads: product pages take many seconds and then fail. The stock service has slowed to 400 ms per call. Little's law says the old setup needed 80 threads on average for 200 requests per second, and a pool of 16 can finish at most 40 per second. The unbounded queue absorbed the difference, so every request got slower, memory climbed with the queue, and the health check, which ran on the same pool, timed out behind the backlog. The checks: queue length and active threads per pool, then the stock service's latency. The mitigation is the release above, or switching the stock call to its fallback value. The lasting fixes are bounded queues, timeouts, a health check that does not share the busy pool, and an alert on queue wait.
Architect's notes
- Name the three limits. For every pool, write down the thread count, the queue bound and what rejection does. If one is "unbounded", say where overload goes instead.
- Separate blocking work. Give each slow dependency its own named executor so that it can exhaust only itself.
- Never rely on the default executor for blocking stages. The common pool is shared by the whole process.
- Timeouts are not cancellation.
orTimeoutfails the future; the work behind it can run on. - Test with latches, not sleeps. A blocked-thread burst gives exact numbers.
Common mistakes
- Using
newFixedThreadPoolon a request path and treating its unbounded queue as a safety margin. - Setting a large maximum with an unbounded queue and expecting extra threads.
- Calling
Future.getorjoininside a task of a pool that must run the task being waited on. - Using
submitfor a task whose failure matters and never reading the future. - Expecting
cancel(true)to interrupt aCompletableFuturetask. - Forgetting shutdown. Fixed-pool threads "will exist until it is explicitly shutdown", so creating a pool per request leaks threads.
Key takeaways
- A pool starts threads up to the core size, then queues, then adds threads up to the maximum when the queue is full, then rejects.
- An unbounded queue means the maximum never matters and overload becomes waiting and memory.
- Average threads in use equals arrival rate times time per request; a pool of N threads and S seconds per request caps throughput at N over S.
CompletableFuturestages without an executor run on the common pool or on the completing thread; pass an executor where it matters.jointhrowsCompletionException,getthrowsExecutionException;cancelandorTimeoutdo not stop the work.- Starvation comes from blocked threads waiting for work the same pool must run; separate pools for blocking work contain it.
Practice exercises
Exercise 1 (20 min). For a service with 120 requests per second and 80 ms per request, compute the average threads busy. State what you would add for bursts and why.
Exercise 2 (45 min). Run the lab. Predict row 1 for a queue of 3, then run it. Explain each number with Figure 1.
Exercise 3 (1 hour). Run "Break it" on the starvation test. Explain why a sleep delays the victim and a wait on the pool's own task can starve it.
Exercise 4 (2 hours). For ShopStream's checkout service (payment, inventory and fraud checks), choose the executors, queue bounds and rejection behaviour for each call, and say which may degrade and which must fail.
Related reading
- The JVM Memory Model and Heap: the happens-before edges that executors and futures provide.
- Garbage Collectors: G1 and ZGC: what an ever-growing queue does to the heap.
- Backpressure: what a rejected task should become for the caller.
- Load Shedding and Retry Storms: refusing work on purpose.
- Timeouts, Retries, Backoff and Jitter: the timeout values that bound a stage.
- Latency vs Throughput: why queue wait is part of latency.
Official documentation
- Java SE 25 API:
ThreadPoolExecutor, the source of the sizing, queueing and rejection rules. - Java SE 25 API:
ExecutorsandExecutorService. - Java SE 25 API:
CompletableFuture. - Java SE 25 API:
ForkJoinPool, the common pool andmanagedBlock.