AI Latency, Token Cost and Failure Capture per Run

What you'll learn: how to record latency, tokens, placeholder-price cost and failures for each evaluated run, why a failed run still costs tokens, and why a lower step limit can look cheaper while answers get worse.

At a glance

Figure 1. What is recorded for one evaluated run

sequenceDiagram accTitle: Recording one evaluated run accDescr: The eval service starts a clock and calls the orchestrator. For each model call the orchestrator goes through the model gateway to the LLM provider, and the gateway adds the reported tokens to a usage meter. The orchestrator returns a response with usage, the clock stops, and the eval service writes a run record with latency, tokens, cost and status to the trace store. participant E as Eval service participant O as Orchestrator participant G as Model gateway participant L as LLM provider participant T as Trace store E->>O: run one question, start clock loop every model call, including a repair retry O->>G: complete messages G->>L: complete messages L-->>G: text and token counts G->>G: add tokens to usage meter G-->>O: completion end O-->>E: response with usage, stop clock E->>T: write run record, latency, tokens, cost, status

How to read this: tokens are counted where every call passes, so the repair retry and tool-planning calls are included.

Record latency, tokens in and out, cost, and how each run ended.
Cost is tokens times a price; the capstone's prices are placeholders.
A failed run still spent tokens. Count them.
A run cut off without an error is degraded, not successful.

Scenario

Composite scenario, not a real company. Lumen Industrial Supply's finance lead asks two questions about the product-discovery assistant: what will it cost per month, and how long will a customer wait? The team has accuracy numbers and nothing else. Someone suggests "just lower the step limit to save money".

What you will be able to do

Say which numbers of a run are measured and which estimated.
Wrap a provider to count tokens for failed runs.
Record a failed or cut-off run without stopping.
Explain why a mean cost hides expensive requests.

Prerequisites

A baseline and two strategies, Calling a provider and AI Architect: Cost Economics for forecasting.

Core ideas

The usage meter

MeteredProvider wraps any provider so UsageMeter records every call, including the repair retry. snapshot turns the totals into the usage object returned with each response.

Verified: capstone/fde-product-discovery-copilot/app/observability/usage.py

    def snapshot(self) -> Usage:
        cost = (self.input_tokens * self._in_price + self.output_tokens * self._out_price) / 1_000_000
        return Usage(
            input_tokens=self.input_tokens,
            output_tokens=self.output_tokens,
            llm_calls=self.llm_calls,
            workflow_steps=self.workflow_steps,
            estimated_cost_eur=round(cost, 6),
        )

Prices come from Settings: 1.0 and 5.0 EUR per million input and output tokens. These are placeholder prices; fill in your provider's published list before quoting a cost. Mock token counts are estimates (words times 1.3).

What the evaluation records, and what it does not

Figure 2. Cost and latency per strategy, 36 questions (from eval/results/latest.json and latest.md)

Strategy	Tokens in / out	Placeholder cost (EUR)	Mean latency (ms)	p95 latency (ms)
baseline	33384 / 3484	0.050804	14.14	25.1
A	30828 / 3283	0.047243	11.905	16.5
B	30680 / 3261	0.046985	16.254	22.6

How to read this: tokens and cost repeat exactly on every run; the two latency columns do not.

The filter removes partner-only chunks and B also drops superseded ones, so the model reads fewer tokens: B costs 7.5% less than the baseline (0.050804 to 0.046985).

Latency is different. A re-run on 2026-11-02 gave means of 21.467, 18.973 and 19.028 ms: the ranking changed. The mock answers in milliseconds, so none of this is model latency.

Failures: the harness stops, the meter forgets

grade catches no exceptions, so one raising question stops the evaluation, and the meter inside run_discovery loses that run's tokens.

Figure 3. How a run ends

flowchart TD accTitle: How one evaluated run ends accDescr: A run either raises an application error or returns. If it raises, record the error class and the tokens counted outside the pipeline, mark it failed, and continue with the next question. If it returns, check the workflow status. Completed or not run means success. Max steps, timeout or planner error means degraded: it finished, but the tool loop was cut off. A["Run one question"] --> B{"Raised an AppError?"} B -- yes --> F["Failed: error class, tokens so far, no question text"] B -- no --> C{"Workflow status"} C -- "completed or not_run" --> S["Success"] C -- "max_steps, timeout, planner_error" --> D["Degraded: answered, but incomplete"] F --> N["Next question"] S --> N D --> N

How to read this: failed and degraded both need a count, because only the first raises.

record_run catches AppError and keeps going.

Verified: capstone/fde-product-discovery-copilot/lessons/M6-L3/m6_l3_run_records.py

def record_run(item: dict[str, Any], strategy: Strategy, svc: Services) -> RunRecord:
    counter = CountingProvider(svc.provider)
    identity = Identity(f"eval-{item['role']}", "eval", item["role"])
    error: str | None = None
    status, failed = "not_run", 0
    latency_ms = 0.0
    try:
        run, latency_ms = timed_run(item["question"], identity, replace(svc, provider=counter), strategy, f"eval-{item['id']}")
        status, failed = run.workflow_status, sum(not s.ok for s in run.steps)
    except AppError as err:
        error, status = type(err).__name__, "error"
    return RunRecord(
        item["id"], strategy.name, error is None, round(latency_ms, 3), counter.calls, counter.input_tokens, counter.output_tokens,
        cost_eur(svc.settings, counter.input_tokens, counter.output_tokens), status, failed, error,
    )

The record keeps the error class, never the question text.

At the customer: a finance lead asks "what does a failed request cost?" A good answer: the tokens spent before it failed, which you count. In the lesson's failure test, one invalid reply plus one repair retry is 2 calls, 20 tokens in and 10 out.

Worked example

Assumptions: placeholder prices of 1.0 and 5.0 per million tokens and the strategy-B totals above.

Mean tokens under B: 30,680 / 36 = 852 in and 3,261 / 36 = 91 out. Cost: (852 x 1 + 91 x 5) / 1,000,000 = 0.001307 per request, or 1.307 per 1,000.

Cost per request (units)	0.001307
Cost per 1,000 requests (units)	1.307

The widget starts at 0.001307 per request. Output is 455 of the 1,307 millionths of a placeholder EUR, 34.8% of the cost, from under 10% of the tokens. The capstone has no cache, so cachedShare is 0.

The mean hides the spread. Question q02 makes 13 model calls (5,398 tokens in, 381 out) and costs 0.007303: 5.6 times the mean.

Build it

Run all 36 questions under strategy B with record_run, then summarise_records. The summary keeps failures in the cost, because you paid for them.

Verified: capstone/fde-product-discovery-copilot/lessons/M6-L3/m6_l3_run_records.py

class CountingProvider:
    """Counts every call and token that passes through, including calls whose reply is later rejected."""

    def __init__(self, inner: LLMProvider) -> None:
        self.name = inner.name
        self._inner = inner
        self.calls = self.input_tokens = self.output_tokens = 0

    def complete(self, messages: Sequence[Message], schema: dict[str, Any] | None = None) -> Completion:
        completion = self._inner.complete(messages, schema)
        self.calls += 1
        self.input_tokens += completion.input_tokens
        self.output_tokens += completion.output_tokens
        return completion

Decision exercise. B has a higher mean latency than A (16.254 against 11.905 ms) and a higher p95 (22.6 against 16.5 ms), with the same retrieval quality and almost the same cost. Choose A, choose B, or wait for more data, and say why.

Model answer

Choose B, and say latency must be re-measured. B removes 2 stale citations and flags both conflict questions; A does not. The gap is plumbing under the mock. Wait only if the customer has a latency requirement you cannot yet test.

Exercise

Run the 36 questions under B with Services.build(Settings(max_steps=3), MockProvider()) and compare with the default. Say whether the lower limit is a saving.

Acceptance criteria:

You report failures, degraded runs, tokens and cost for both limits.
Your conclusion names what you traded for the saving.

Done when your answer compares the cost fall with the number of runs that did not finish.

Hint 1: failures are not the only bad outcome

Check summarise_records(...)["degraded"]: a run at the limit still returns.

Hint 2: compute the saving as a share

Divide the capped cost by 0.046985, subtract from 1, and compare with the share of the 23 tool-using runs cut off.

Common mistakes

Reporting a mean only. One request can cost five times the mean.
Calling a cut-off run a success. Count it as degraded.
Hard-coding a price. Use the published list and record the date.
Quoting mock latency as a promise.

Knowledge check

1. Why count tokens in a wrapper around the provider instead of only from the final response?

A request makes several calls, including a repair retry, and only a wrapper sees them all. A failed run returns no response to read from.

2. Why is the cost in the capstone not a real cost?

It multiplies estimated token counts (words times 1.3) by placeholder prices. The shape carries over to real prices; the values do not.

3. Strategy B costs 7.5% less than the baseline. Where does the saving come from?

The filter cuts input tokens from 33,384 to 30,828 (A), and dropping superseded datasheets trims them to 30,680 (B). It is a side effect of the safety fixes.

4. Lowering the step limit cut cost by 27.6% with 0 failures. Why is that not a saving?

Seven of the 23 tool-using runs ended with status max_steps, so their answers rest on an incomplete search. Nothing raised; only a degraded count shows it.

5. Why store the error class and not the question text in a failure record?

The text may contain customer data, and failure records travel further than responses. Class, status and tokens suffice to debug and cost it.

You can now

Say which usage numbers are measured and which estimated.
Count tokens for failed runs.

AI Latency, Token Cost and Failure Capture per Run

At a glance

Scenario

What you will be able to do

Prerequisites

Core ideas

The usage meter

What the evaluation records, and what it does not

Failures: the harness stops, the meter forgets

Worked example

Build it

Exercise

Common mistakes

Knowledge check

You can now

Further reading

Next

Enroll to keep reading

Ask about this article

Enjoyed this? Get more like it
every Monday.

Comments

At a glance

Scenario

What you will be able to do

Prerequisites

Core ideas

The usage meter

What the evaluation records, and what it does not

Failures: the harness stops, the meter forgets

Worked example

Build it

Exercise

Common mistakes

Knowledge check

You can now

Further reading

Next

Enroll to keep reading

Ask about this article

Enjoyed this? Get more like it every Monday.

Comments

Enjoyed this? Get more like it
every Monday.