What you'll learn: what changes when the model is real, how to repeat one request 20 times and count valid replies, failures, latency and tokens, and how to price a usable answer. This page has no real-model numbers: you measure your own.
At a glance
Figure 1. The repeat-run harness
flowchart LR
accTitle: Repeat-run harness
accDescr: One prompt goes into a loop of n calls. Each call goes to the provider directly. If the call raises, the exception type is counted. If it returns text, the text is parsed and validated; a valid reply is counted and stored, an invalid one is counted by kind with a short sample. Every call also records seconds; a call that returns text also records tokens. All counts end in one result object.
P["One prompt"] --> L["Loop: n calls"]
L --> C["provider.complete"]
C -->|"raised"| E["Count provider_error plus type name"]
C -->|"text"| V{"Parse and validate"}
V -->|"valid"| O["Count valid, keep canonical output"]
V -->|"invalid"| F["Count not_json or wrong_shape, keep a sample"]
C -.->|"every call"| T["Record seconds, and tokens if text came back"]
E --> R["Result object"]
O --> R
F --> R
T --> R
How to read this: nothing retries or repairs, so every count describes the first reply.
The mock never fails or varies. A real model can do both, and calls take seconds.
Repeat each prompt 20 times and count the outcomes.
Call the provider directly: the repair retry hides failures.
Cost per usable answer is total cost over valid replies, so failures raise it.
Scenario
Composite scenario, not a real company. Lumen Industrial Supply's team has run the copilot on the mock for weeks; every check passed. A real provider key has arrived. The lead asks how often the model returns JSON the code accepts, whether one request gives the same answer twice, and how long a call takes. Worked-example numbers are illustrative.
What you will be able to do
Say what changes between the mock and a real provider.
Run run_repeated and fill a results table with your own numbers.
Compute cost per valid answer from recorded usage.
Predict first. One prompt runs 20 times through the full pipeline, repair retry included, and all 20 answers are valid. Can you say the model returned valid JSON first try 20 times?
Answer
No, you cannot tell. The repair retry may have fixed bad first replies. The lab calls provider.complete directly.
Count the first reply
Each call is counted as provider_error: plus the exception type, not_json, wrong_shape or valid. Replies that differ only in whitespace or a Markdown fence parse to the same object: one distinct answer.
What the provider documents (as of 2026-11-03, check each at its source)
The Python SDK reads ANTHROPIC_API_KEY, retries connection errors, 408, 409, 429 and 5xx twice by default, and times out after 10 minutes (Python SDK). A slow call may be three attempts.
The API names errors: 429 rate_limit_error, 500 api_error, 504 timeout_error, 529 overloaded_error (API errors). The SDK raises classes such as RateLimitError (429), AuthenticationError (401) and APITimeoutError; the lab counts the class name (SDK errors).
A reply that reaches max_tokens has stop reason max_tokens (stop reasons). The adapter sets 2,048 and returns only text, so a cut-off reply shows as not_json.
The provider states that every Claude API model ID is a pinned snapshot (models overview): record it.
A Structured Outputs feature is documented with "guaranteed schema compliance" (output consistency); the adapter does not use it.
Keep the key safe
The key guidance: use environment variables, ignore .env files in git, use separate keys per environment, scan for committed secrets in CI. The capstone reads only real environment variables.
When validation fails: first checks
Figure 2. First check for each failure kind
flowchart TD
accTitle: First checks by failure kind
accDescr: A failure is either a provider error, not_json, or wrong_shape. For a provider error, read the exception type: rate limit, timeout, or authentication. For not_json, read the printed sample: prose before the JSON, a reply cut off mid-object, or empty text. For wrong_shape, read which field failed validation and whether the schema or the prompt is wrong.
F["A failure in your table"] --> K{"Which kind?"}
K -->|"provider_error"| P["Read the type name: rate limit, timeout, authentication"]
K -->|"not_json"| N["Read the sample: prose first, cut off mid-object, or empty"]
K -->|"wrong_shape"| W["Read the validation error: which field, wrong type or out of range"]
N --> M["Cut off: check max_tokens"]
N --> Q["Prose first: tighten the prompt, then re-measure"]
W --> S["Wrong schema or wrong prompt: change one, re-measure"]
How to read this: read a real sample first, and change one thing per re-run.
Your table shows
Decision for the Lumen pilot
Valid rate at or above your gate on every prompt
Keep the repair retry as a safety net; re-run another day
Valid rate below the gate on any prompt
No pilot yet: use Figure 2, change one thing
More than one distinct valid answer
Compare the differing fields; if one sales engineers read changes, add the question to the golden set
Assumptions (illustrative, not a model): a scripted provider plays 20 hand-written outcomes: 13 identical valid replies, 2 in a Markdown fence, 1 valid with an extra keyword, 2 with prose before the JSON, 1 with quantity 0 (the schema requires above 0) and 1 timeout. The latencies are made up; the timeout is the 7th call, 8 seconds.
defclassify(model: type[BaseModel], text: str) -> str | None:
"""None when the reply is valid, otherwise the failure kind."""try:
parse_json_model(model, text)
except json.JSONDecodeError: # a ValueError subclass, so catch it before anything broaderreturn NOT_JSON
except ValidationError:
return WRONG_SHAPE
returnNone
Run it with PYTHONPATH=. ./.venv/Scripts/python lessons/M2-L8/m2_l8_variability.py --demo. Real output:
| Prompt | Valid | Failures | Distinct valid | Median s | p95 s | Max s | Tokens in | Tokens out |
|---|---|---|---|---|---|---|---|---|
| scripted (illustrative) | 16/20 | not_json x2, provider_error:TimeoutError x1, wrong_shape x1 | 2 | 1.00 | 1.30 | 8.00 | 1463 | 631 |
Quantity
Calculation
Result
Distinct valid answers
15 identical or fenced, 1 with an extra keyword
2
p95 (nearest rank)
rank 19 of 20 sorted latencies
1.30 s
Cost of the run
(1,463 x 1 + 631 x 5) / 1,000,000
0.004618
Cost per valid answer
0.004618 / 16
0.000289
Same tokens, if all 20 were valid
0.004618 / 20
0.000231
Prices (1 and 5 per million tokens) are the capstone's placeholders. The timeout returned no tokens; the three invalid replies did. Failures made each usable answer 25 percent dearer (20 / 16 = 1.25). The median hides the 8-second call; the max shows it.
At the customer: "How reliable is it?" Show your repeated-run table for their request types, with model ID and date, and what the code does per failure kind.
Build it
Code: capstone/fde-product-discovery-copilot/lessons/M2-L8/. Step 1: the loop calls the provider once per repeat and files each outcome.
schema = model_type.model_json_schema()
result = RepeatResult(n=n)
for _ inrange(n):
started = clock()
try:
completion = provider.complete(messages, schema)
except Exception as err: # noqa: BLE001 - any provider failure is a data point
result.latencies_s.append(clock() - started)
# The type name only. An exception message can echo a request, so it is not kept.
result.failures[f"{PROVIDER_ERROR}:{type(err).__name__}"] += 1continue
result.latencies_s.append(clock() - started)
result.input_tokens += completion.input_tokens
result.output_tokens += completion.output_tokens
kind = classify(model_type, completion.text)
if kind isNone:
result.valid += 1
parsed = parse_json_model(model_type, completion.text)
result.valid_outputs[json.dumps(parsed.model_dump(), sort_keys=True)] += 1else:
result.failures[kind] += 1iflen(result.failure_samples) < max_samples:
result.failure_samples.append((kind, completion.text[:sample_chars]))
return result
Step 2: tests that it keeps no exception message and never retries.
Step 4: the lab. First write your gate: the lowest valid rate you accept per prompt. Then set the variables from your provider's documentation (the repository fixes no model ID). read -s keeps the key out of shell history.
export LLM_PROVIDER=anthropic
read -rs -p "API key: " ANTHROPIC_API_KEY; export ANTHROPIC_API_KEY
export ANTHROPIC_MODEL="<model id from the provider's models page>"export PRICE_IN_PER_MTOK="<input price from the provider's price list>"export PRICE_OUT_PER_MTOK="<output price from the provider's price list>"
PYTHONPATH=. ./.venv/Scripts/python lessons/M2-L8/m2_l8_variability.py --n 20 --price-in "$PRICE_IN_PER_MTOK" --price-out "$PRICE_OUT_PER_MTOK"
The script prints the table, up to three failing replies per prompt and the cost lines. Copy your numbers here with date, model ID and gate:
Prompt
Valid (of 20)
Failures by kind
Distinct valid
Median s
p95 s
Max s
Cost per valid answer
pumps
gateway
logger
Apply the decision table to each prompt, then try:
Read the lab results (interactive when JavaScript is on)
The full pipeline, repair retry included, gives 20 of 20 valid. A manager wants to quote 100 percent. What do you do?
Quote 100 percent
Re-run with direct provider calls and quote the first-reply rate
Quote the average of the three prompts
Answer
Re-run with direct provider calls and quote the first-reply rate. The repair retry can fix a bad first reply, hiding the first-reply failure rate.
Three of 20 replies fail as not_json; the samples end mid-object. What do you check first?
Whether max_tokens cut the replies off
Whether the prompt asks for JSON clearly
Whether the schema has too many fields
Answer
Whether max_tokens cut the replies off. Mid-object endings point to max_tokens (Figure 2); a schema problem shows as wrong_shape.
A cheaper model has a lower valid rate. How do you compare cost?
Price per million tokens
Median latency only
Cost per valid answer
Answer
Cost per valid answer. Invalid replies still carry tokens, so a cheap, failing model can cost more per usable answer.
Exercise
Write meets_gate(results, min_valid_rate): it takes what run_lab returns and returns the labels of prompts below the gate.
Acceptance criteria: with a gate of 0.95, 19 valid of 20 passes and 18 fails; the check is per prompt; an empty list means all passed. Done when your scripted-provider tests pass.
Hint 1: one number or three?
An average can hide a bad prompt. Use each valid_rate.
Hint 2: the shape
Loop over results.items(); keep labels below the gate.
Common mistakes
Measuring through the repair retry. It hides first-reply failures.
One run, one conclusion. Re-run another day.
Averaging across prompts. One bad prompt hides in the mean.
Quoting the mock. It measures your pipeline, not a model.
Knowledge check
1. Why does the lab call `provider.complete` instead of `complete_structured`?
complete_structured repairs a bad reply once, hiding first-reply failures.
2. Why does the harness keep the exception type but not its message?
The type says what happened; a message can echo private request text.
3. Two valid replies differ only in a Markdown fence. How many distinct answers is that?
One. Both parse to the same object; the harness compares the parsed result.
4. A measured latency is longer than one attempt should take. What might explain it?
The SDK retries some errors twice: one measured call can be three attempts.
5. Why is cost per valid answer higher than cost per call?
Invalid replies still returned tokens: they are in the total but not in the usable count.
You can now
Explain what a mock hides and a real provider adds.
Run the repeat harness and compute cost per valid answer.
Lesson 7 of 7 · Forward Deployed Engineer: Build LLM Features
Run the capstone against a real provider, repeat each prompt 20 times, and record valid replies, failure kinds, latency and tokens in your own table.
In the rest of this lesson
Build it
Exercise
Common mistakes
Knowledge check
You can now
Further reading
Enrolling is free. It unlocks every lesson in Forward Deployed Engineer: Build LLM Features, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.
Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.
Get new articles by email
Free. We email a confirmation link first, and you can unsubscribe from any email.