What you'll learn: what the caller and the Python service each promise at the integration boundary, how a retry with an idempotency key replays a stored answer instead of running the work twice, what the capstone's in-memory store does not do, and how to choose between a synchronous answer and an asynchronous job.
At a glance
Figure 1. A lost reply, retried with the same key
How to read this: the second request does not start the pipeline; the key and body together show it is a retry of finished work.
- The boundary is a contract: the caller promises a deadline and a key, the service a replay.
- A POST is safe to retry with an idempotency key.
- Bound retries by count and deadline, and add jitter.
- The capstone's store is in memory; production needs a shared one.
Scenario
Composite scenario, not a real company. At Lumen Industrial Supply, the fictional distributor, the sales portal calls the discovery service, and a call takes seconds. Sales engineers double-click and a flaky network drops replies. Last week one request ran three times and the team paid for three model calls. The portal team asks for retries; the service owner asks what a retry must not do. Numbers are illustrative.
What you will be able to do
- Write the boundary contract for caller and service.
- Predict a retry's result for the same key and body, a different body, and no key.
- Compute how many attempts a deadline allows.
- Choose between a synchronous answer and an asynchronous job.
Prerequisites
Async, HTTP clients, timeouts and retries and Reliability and scaling.
Core ideas
The boundary is a contract
Predict first. A caller sends a request with Idempotency-Key: k1. The service finishes, but the reply is lost, so the caller retries with the same key and body. What comes back, and how often did the pipeline run?
Answer
The stored response, with the header Idempotent-Replay: true (defined by this capstone, not a standard). The pipeline ran once.
| Side | Promises |
|---|---|
| Caller | A deadline on every call; one key per user action, reused on retries; retry only 408, 429, 502, 503 and 504; a bounded number of attempts |
| Service | Same key and body, same response; same key, different body, refused (409); one error shape with a trace id |
RFC 9110 says PUT, DELETE and safe methods are idempotent, and that a client should not automatically retry a request with a non-idempotent method unless it can know the request is effectively idempotent or was never applied. POST is not in that list; an idempotency key is that means.
The Idempotency-Key header is a draft, not a standard
As of 2026-11-02 the header is described by an Internet-Draft, draft-ietf-httpapi-idempotency-key-header, an active draft of the IETF httpapi working group (revision 08, updated 2026-10-22, expiring 25 April 2027), not an RFC.
The draft suggests 409 when a retry arrives while the first request still runs and 422 when a key is reused with a different payload. The capstone answers 409 for a different payload and has no "still running" case.
What the capstone's store does
Verified: capstone/fde-product-discovery-copilot/app/idempotency.py
def get(self, identity: Identity, key: str, body: str) -> dict[str, object] | None:
"""Keys are scoped to tenant and user, so one caller can never read another's response."""
with self._lock:
entry = self._items.get((identity.tenant, identity.user_id, key))
if entry is None:
return None
if entry[0] != self._fingerprint(body):
raise IdempotencyConflict("This Idempotency-Key was already used with a different request.")
return entry[1]The store is keyed by tenant, user and key, and keeps a hash of the request body next to the response. Its limits (tests in lessons/M5-L3):