Async, HTTP Clients, Timeouts and Retries in Python

Retry-After is a number of seconds or an HTTP date. The lesson's code reads the seconds form and ignores the date form.

Backoff with jitter, written down

In code: the wait before each retry is random between 0 and a window that doubles each time, up to a cap (full jitter).

Verified: capstone/fde-product-discovery-copilot/lessons/M1-L4/m1_l4_retry.py

@dataclass(frozen=True)
class RetryPolicy:
    max_attempts: int = 3  # the first try plus two retries
    base_s: float = 0.2
    cap_s: float = 2.0

    def window(self, retry_index: int) -> float:
        """Upper bound of the wait before retry number `retry_index` (0 is the first retry)."""
        return min(self.cap_s, self.base_s * 2**retry_index)

    def backoff(self, retry_index: int, rng: random.Random) -> float:
        """Full jitter: a uniform random wait between 0 and the window."""
        return rng.uniform(0, self.window(retry_index))

Without jitter, clients that failed together retry together and hit the recovering server in one spike. A lesson test draws 500 waits per row and checks that all stay inside the window and reach into its top tenth.

One deadline for the whole call

A timeout limits one attempt. Three attempts of 1 second plus waits can keep the caller waiting over 3 seconds without any timeout being exceeded. The fix is a deadline: a time budget every attempt and wait draws from.

Verified: capstone/fde-product-discovery-copilot/lessons/M1-L4/m1_l4_retry.py

@dataclass
class Deadline:
    """A time budget that every attempt and every wait draws from."""

    total_s: float
    clock: Callable[[], float] = time.monotonic
    started: float = field(init=False)

    def __post_init__(self) -> None:
        self.started = self.clock()

    def remaining(self) -> float:
        return max(0.0, self.total_s - (self.clock() - self.started))

The clock argument lets tests pass a fake clock and never wait, as the capstone's WorkflowRunner does.

Figure 2. One call with a 3.0 second deadline and a 1.0 second attempt timeout (illustrative)

How to read this: each attempt asks the deadline how long it may take, and a wait that would pass the deadline is never started.

Verified: capstone/fde-product-discovery-copilot/lessons/M1-L4/m1_l4_retry.py

    last = "no attempt made"
    for attempt in range(policy.max_attempts):
        left = deadline.remaining()
        if left <= 0:
            raise DeadlineExceeded(f"no time left before attempt {attempt + 1}")
        hinted: float | None = None
        try:
            response = await client.get(url, timeout=min(attempt_timeout_s, left))
            if response.status_code not in RETRYABLE_STATUS:
                return response
            last = f"status {response.status_code}"
            hinted = retry_after_seconds(response)
        except httpx.TransportError as exc:  # timeouts and connection failures
            last = type(exc).__name__
        if attempt == policy.max_attempts - 1:
            break
        wait = policy.backoff(attempt, rng) + (hinted or 0.0)  # Retry-After is a floor, jitter goes on top
        if wait >= deadline.remaining():
            raise DeadlineExceeded(f"a wait of {wait:.2f}s would pass the deadline (last: {last})")
        await sleep(wait)
    raise RetriesExhausted(f"{policy.max_attempts} attempts failed (last: {last})")

Note three details: no wait after the last attempt; the attempt timeout shrinks to the time left; and two errors, RetriesExhausted (upstream failing, answer 502) and DeadlineExceeded (budget gone, answer 504).

Retries stack, and the load multiplies

If the browser, your service and the SDK under it all retry, the layers multiply. Set layers to 3 and attempts to 3 in the widget.

The widget's start values give 27 attempts per call, 108 per second at 4 calls per second. Give retries to one layer and switch the others off: the reliability lesson calls this "one retry owner".

Worked example

Lumen's stock call uses the default policy (3 attempts, base 0.2 s, cap 2.0 s), a 1.0 s attempt timeout, and a 3.0 s deadline. The upstream hangs on every attempt. Waits take the full window, which is the worst case.

Assumptions: all values illustrative; the upstream never answers; jitter draws the longest wait each time.

• Without the deadline the same policy can take 3 x 1.0 + 0.2 + 0.4 = 3.6 seconds, which the method worst_case_s returns.
• With the 3.0 s deadline the third attempt is cut to 0.4 s.
• If the upstream answers 503 with Retry-After: 30 and the deadline is 5 s, the code raises DeadlineExceeded at once: a 30 second wait could never fit.

At the customer: "What will the engineer see when the ERP is down?" A good answer: an honest "stock unavailable, last checked at ..." within the deadline, never a spinner that ends in a timeout. Agree the deadline users tolerate, then fit attempt timeouts inside it.

Build it

Everything runs offline. From capstone/fde-product-discovery-copilot:

./.venv/Scripts/python -m pytest lessons/M1-L4

No test opens a socket. httpx.MockTransport takes a handler that returns a response or raises an httpx exception. This test makes attempt 1 time out and attempt 2 succeed:

Verified: capstone/fde-product-discovery-copilot/lessons/M1-L4/test_m1_l4_retry.py

def test_a_timeout_raised_by_the_transport_is_retried() -> None:
    state = {"n": 0}

    def flaky(request: httpx.Request) -> httpx.Response:
        state["n"] += 1
        if state["n"] == 1:
            raise httpx.ReadTimeout("slow upstream", request=request)
        return httpx.Response(200, json={"qty": 1})

    go, calls, _ = run(flaky)
    assert asyncio.run(go()).status_code == 200 and len(calls) == 2

1. Run the suite and read the test names: they are the specification.
2. In m1_l4_retry.py drop 502 from RETRYABLE_STATUS and see which tests fail.
3. In m1_l4_gateway.py follow how RetriesExhausted becomes 502 and DeadlineExceeded 504.

Exercise

Decision exercise. Lumen's ERP has two endpoints. GET /stock/{sku} is read-only. POST /orders creates an order and accepts an optional Idempotency-Key header. One call in five times out. Choose a policy and justify it:

1. Retry every call up to 5 times, immediately.
2. Retry GET up to 3 times with backoff and jitter; retry POST only with an idempotency key; both under a deadline.
3. Never retry; fail fast.

The task. Write a test (no new production code) proving a call never outlasts its deadline when the upstream never answers.

Acceptance criteria:

• The transport handler advances a fake clock by the attempt's timeout, then raises httpx.ReadTimeout.
• The test is parametrized over at least four deadlines (for example 0.5, 1.0, 3.0 and 10.0 seconds) and several random seeds.
• It asserts that the fake clock never ends after the deadline.

Done when pytest passes for every deadline and seed, and fails if you change min(attempt_timeout_s, left) in get_with_retries to attempt_timeout_s.

Common mistakes

• A new client per request. The pool is never used. Create one client at start-up.
• time.sleep or a blocking library inside async def. It freezes every request. Use await asyncio.sleep, an async client or plain def.
• Timeouts without a deadline. Three 1-second attempts and their waits outlast the caller. Budget the whole call.
• Retrying non-idempotent requests blindly. A retried POST can create two orders. Send an idempotency key or do not retry.

Knowledge check

You can now

• Decide between async def, def and a worker from what the handler waits on.
• Share one httpx.AsyncClient and name the four timeouts.
• Write a retry loop with backoff, jitter, Retry-After and a deadline.
• Say which requests are safe to retry.
• Test retries offline with MockTransport and a fake clock.

Further reading

• httpx: Timeouts and Async support, the client used here.
• FastAPI: Concurrency and async / await, when to use async def or def.
• RFC 9110: HTTP Semantics, idempotent methods and Retry-After.

Last verified: 2026-11-02

Next

Errors, configuration and logging