LLM Provider Calls: Adapter, Mock, Rate Limits, Cost

What you'll learn: how to put every model call behind one small contract, why the capstone defaults to a mock, how to retry with a deadline without multiplying retries, and how to record tokens and cost per request.

At a glance

Figure 1. The provider layer: one contract, wrappers around it

How to read this: every caller sees only complete(), so the counting, the retries and the choice between a mock and a real provider can change without touching the pipeline (app/ has the meter and both providers; ResilientProvider is built in this lesson).

All model calls go through one method, LLMProvider.complete, so providers are swappable and testable.
The mock is a rule-based stand-in for a model. It makes the course runnable offline; it is not evidence about a real model.
A rate limit (429) or an overload is normal traffic, not an exception. Retry a few times, with a deadline.
Exactly one layer owns retries. The official SDK retries by default, so a wrapper must switch that off.
One discovery request makes nine model calls, so count tokens per request, not per call.

Scenario

Composite scenario, not a real company. Lumen Industrial Supply, the fictional distributor, puts the copilot in front of its sales engineers. On Monday mornings about 30 of them ask a question within the same minute. The demo runs on the mock, and a key for a real provider arrives this week. What breaks first at 30 requests a minute, how long may a request retry, and what does one request cost? All numbers are illustrative.

What you will be able to do

Explain what the LLMProvider contract hides and why the pipeline never imports a vendor SDK.
Wrap a provider with bounded retries, a server-given wait and a deadline, and prove the behavior in a test without sleeping.
Name the three timeouts in a model call and say which one each layer enforces.
Compute the token cost of one request from recorded usage and a placeholder price.

Prerequisites

Async, HTTP clients, timeouts and retries covers the general mechanics; Structured outputs and schema validation uses the same complete method. For why a retry has one owner and how jittered backoff works, read Reliability and scaling; for the cost model behind the widget, Cost economics.

Core ideas

One contract for every provider

Piece	What it is	Why it exists
`Message`	role plus text	The same input for every provider
`Completion`	text plus input and output token counts	The only thing the pipeline reads back
`LLMProvider`	a protocol with one method, `complete`	Swap, wrap and fake providers without changing callers

Verified: capstone/fde-product-discovery-copilot/app/llm/base.py

@dataclass(frozen=True)
class Message:
    role: Literal["system", "user", "assistant"]
    content: str


@dataclass(frozen=True)
class Completion:
    text: str
    input_tokens: int
    output_tokens: int


class LLMProvider(Protocol):
    name: str

    def complete(self, messages: Sequence[Message], schema: dict[str, Any] | None = None) -> Completion:
        """Return the model's text. If `schema` is given the text must be JSON matching it."""
        ...

The contract is a Python Protocol: any class with a name and a complete method fits, with no inheritance (think of a small Java interface). It has no streaming, no tool blocks and no stop reason; Streaming and invalid output comes back to that gap.

The mock is a stand-in, not a model

Verified: capstone/fde-product-discovery-copilot/app/llm/mock.py

class MockProvider:
    name = "mock"

    def complete(self, messages: Sequence[Message], schema: dict[str, Any] | None = None) -> Completion:
        task = task_of(messages)
        first_user = next(m.content for m in messages if m.role == "user")
        if task == TASK_EXTRACT:
            payload: dict[str, Any] = extract_requirements(first_user).model_dump()
        else:
            payload = _HANDLERS[task](json.loads(first_user))
        text = json.dumps(payload, sort_keys=True)
        return Completion(text, sum(estimate_tokens(m.content) for m in messages), estimate_tokens(text))

MockProvider reads the task name from the system message, runs a rule (regular expressions, a fixed script, sentence copying) and reports estimated token counts. It is deterministic, free and fast, so the test suite runs offline. It never rate-limits, times out or returns malformed JSON; a real model does all of these, so every failure in this lesson has to be simulated in tests.

WARNING

Do not quote a result produced with the mock as a statement about model quality. It measures your pipeline, not a model.

What goes wrong on the way to a real provider

Failure	What the provider documents	What your code should do
429 `rate_limit_error`	A limit was hit; the response may carry a `retry-after` header	Wait that long, then retry within the deadline
429 for a tier spend cap	No `retry-after`; keeps failing until access resumes	Do not retry; alert a person
5xx, 529 `overloaded_error`	Internal error or temporary overload	Retry with backoff
504 or no answer	The request timed out	Retry once or twice, then fail clearly
400, 401, 403	The request or the key is wrong	Never retry; fix the cause

Per the provider's documentation (checked 2026-11-02), limits are measured in requests, input tokens and output tokens per minute, and capacity is replenished continuously (a token bucket). A burst can trip a limit even when the per-minute average is below it. Your own limits depend on your account tier; read them in the provider's console.

Figure 2. One rate-limited call, retried inside the deadline

sequenceDiagram accTitle: A rate-limited call retried once accDescr: The Orchestrator calls the ResilientProvider, which calls the LLM provider. The provider answers 429 with a retry-after of 7 seconds. The ResilientProvider waits 7 seconds, which still fits the 10 second deadline, calls again and receives a completion with token counts. The Orchestrator sees one completion and the meter records one call. participant O as Orchestrator participant R as ResilientProvider participant P as LLM provider O->>R: complete(messages) R->>P: attempt 1 P-->>R: 429 with retry-after 7 Note over R: 7 s wait fits the 10 s deadline R->>P: attempt 2 P-->>R: 200 with text and usage R-->>O: Completion Note over O: the meter records one call

How to read this: the retry is invisible to the caller, except that the call took seven seconds longer.

Three timeouts, three owners

Timeout	Who enforces it	Default in this stack
One attempt	The HTTP client inside the SDK (`timeout=`)	The Python SDK documents 10 minutes: far too long for a person waiting
The whole call, retries included	`ResilientProvider` (`deadline_s`)	10 seconds, set by you
The whole workflow	`WorkflowRunner` (`workflow_timeout_s`)	5 seconds, checked only between steps

The deadline stops new attempts; it cannot interrupt one already running, so the per-attempt timeout must be shorter than the deadline. The workflow limit works the same way: the runner checks the clock before each step, so one slow call can pass it.

Decision exercise. The official Python SDK retries connection errors, 408, 409, 429 and 5xx responses twice by default. You wrap the provider in your own three-attempt retry. A teammate says "more retries, more resilience, leave both on". Choose: (a) leave both on, (b) set the SDK to max_retries=0 and keep yours, (c) keep the SDK's retries and delete yours. Justify in two sentences.

Model answer

Choose (b), or (c) if you need no deadline or per-error rules. With both on, one logical call can make 3 x 3 = 9 HTTP attempts, each with its own timeout, and nothing bounds the total. Pick (b) here, because your layer knows the deadline, the usage meter and the rule "never retry a 400"; the SDK knows none of those.

Recording usage

Verified: capstone/fde-product-discovery-copilot/app/observability/usage.py

    def record(self, completion: Completion) -> None:
        self.input_tokens += completion.input_tokens
        self.output_tokens += completion.output_tokens
        self.llm_calls += 1

    def snapshot(self) -> Usage:
        cost = (self.input_tokens * self._in_price + self.output_tokens * self._out_price) / 1_000_000
        return Usage(
            input_tokens=self.input_tokens,
            output_tokens=self.output_tokens,
            llm_calls=self.llm_calls,
            workflow_steps=self.workflow_steps,
            estimated_cost_eur=round(cost, 6),
        )

record adds the token counts from each Completion; snapshot turns them into the Usage object returned with every API response. The prices are placeholders (1.0 and 5.0 per million tokens in the capstone), never a provider's real price. MeteredProvider calls record after every successful complete:

Verified: capstone/fde-product-discovery-copilot/app/observability/usage.py

class MeteredProvider:
    """Wraps any provider so every call is counted, including the repair retry."""

    def __init__(self, inner: LLMProvider, meter: UsageMeter) -> None:
        self.name = inner.name
        self._inner = inner
        self._meter = meter

    def complete(self, messages: Sequence[Message], schema: dict[str, Any] | None = None) -> Completion:
        completion = self._inner.complete(messages, schema)
        self._meter.record(completion)
        return completion

A call that raises (a timeout, a 429) never reaches record, so retries do not inflate the count, and the meter cannot show wasted work. Whether a provider bills for an abandoned request is the provider's rule; read it before you promise a customer a cost ceiling.

Worked example

Three numbers explain most of the Monday-morning question. Assumptions: 30 requests in the same minute; nine model calls per request (the count measured below); an illustrative limit of 200 requests per minute for the whole organization; placeholder prices of 1 and 5 units per million input and output tokens.

Quantity	Calculation	Result
Model calls in that minute	30 requests x 9 calls	270
Calls over the illustrative limit	270 - 200	70
Input tokens in that minute	30 x 2,792	83,760
Output tokens in that minute	30 x 296	8,880
Cost of one request	(2,792 x 1 + 296 x 5) / 1,000,000	0.004272 units

The request count is not the call count: 30 requests that look safe against a limit of 200 become 270 calls. Merging steps reduces calls; retries cannot fix a sustained excess, they only delay it. The usage figures come from the capstone's pumps request on the mock: 9 calls, 2,792 input tokens, 296 output tokens. They are estimates (words times 1.3), not a tokenizer's counts.

Enter them in the widget. Output tokens cost five times as much here, yet input dominates the bill because there are nearly ten times as many.

Cost per request (units)	0.004272
Cost per 1,000 requests (units)	4.272

At the start values the widget gives 0.004272 units per request and 4.272 per 1,000. Input costs 2,792 units of the 4,272 (65.4 percent) and output 1,480 (34.6 percent). Set cachedShare to 0.5 for a what-if on caching a shared prompt prefix; the capstone does not cache. Fill in your provider's published prices before quoting money to a customer.

At the customer: "What happens when the provider is down or slow?" A good answer names the failure kinds, says which are retried, gives the deadline the user will wait, and says what the user sees after it (a clear error, not a hang or a made-up answer).

Build it

The code is in capstone/fde-product-discovery-copilot/lessons/M2-L4/. Step 1: define a closed set of provider errors, so the rest of the code never catches a vendor exception.

Verified: capstone/fde-product-discovery-copilot/lessons/M2-L4/m2_l4_resilient.py

class ProviderError(Exception):
    """Base class: something went wrong talking to the model provider."""


class ProviderRateLimited(ProviderError):
    def __init__(self, message: str = "rate limited", retry_after: float | None = None) -> None:
        super().__init__(message)
        self.retry_after = retry_after


class ProviderUnavailable(ProviderError):
    """A 5xx, an overload or a dropped connection: worth retrying."""


class ProviderTimeout(ProviderError):
    """One attempt took longer than its per-attempt timeout."""

Step 2: write the retry loop. The last attempt re-raises, and so does any wait that would pass the deadline.

Verified: capstone/fde-product-discovery-copilot/lessons/M2-L4/m2_l4_resilient.py

    def _wait_for(self, attempt: int, err: ProviderError) -> float:
        if isinstance(err, ProviderRateLimited) and err.retry_after is not None:
            return err.retry_after  # the server said exactly how long: earlier retries would fail
        return self._jitter(min(self._max_delay, self._base * 2 ** (attempt - 1)))

    def complete(self, messages: Sequence[Message], schema: dict[str, Any] | None = None) -> Completion:
        started = self._clock()
        for attempt in range(1, self._max_attempts + 1):
            try:
                return self._inner.complete(messages, schema)
            except RETRYABLE as err:
                wait = self._wait_for(attempt, err)
                out_of_time = self._clock() - started + wait > self._deadline
                if attempt == self._max_attempts or out_of_time:
                    raise  # fail fast: the caller gets the real error, not a silent delay
                self._sleep(wait)
        raise AssertionError("unreachable")  # pragma: no cover

Step 3: prove the deadline rule without sleeping. The test fakes the clock and sleep, so the waits are exact.

Verified: capstone/fde-product-discovery-copilot/lessons/M2-L4/test_m2_l4_resilient.py

def test_a_wait_that_would_pass_the_deadline_fails_fast() -> None:
    t, inner = FakeTime(), Flaky(ProviderRateLimited(retry_after=30.0))
    with pytest.raises(ProviderRateLimited):
        resilient(inner, t, deadline_s=10.0).complete(MSG)
    assert inner.attempts == 1 and t.waits == []  # it did not sleep 30 s to find out

Step 4: map the official SDK's exceptions onto your errors. The timeout error is a subclass of the connection error, so test it first. The function keeps the retry-after header when there is one. It treats every 429 as retryable, so a spend-cap 429 would burn its bounded attempts; the provider documents error.details.error_code as enforced_spend_limit_reached for that case, so check it before mapping (untested: the adapter never ran live).

Verified: capstone/fde-product-discovery-copilot/lessons/M2-L4/m2_l4_resilient.py

def classify_anthropic_error(err: Exception) -> ProviderError | None:
    """Map the official SDK's typed exceptions onto ours. None means: do not retry, let it propagate."""
    import anthropic

    if isinstance(err, anthropic.RateLimitError):
        raw = err.response.headers.get("retry-after")
        return ProviderRateLimited(str(err), float(raw) if raw and raw.replace(".", "", 1).isdigit() else None)
    if isinstance(err, anthropic.APITimeoutError):  # a subclass of APIConnectionError: check it first
        return ProviderTimeout(str(err))
    if isinstance(err, (anthropic.APIConnectionError, anthropic.InternalServerError)):
        return ProviderUnavailable(str(err))
    return None

Step 5: stack the wrappers so each does one job: error mapping inside retry, the meter outside both. max_retries=0 and timeout are options of the official client (anthropic.Anthropic(...)); the capstone's AnthropicProvider uses the SDK defaults and takes neither, so you would add them there. The pseudocode shows intent, not the real constructor. That adapter has never run against the live API in this build, so test it with your own key.

Pseudocode.

MeteredProvider(
  ResilientProvider(
    ErrorMappingProvider(AnthropicProvider(max_retries=0, timeout=8))))

Step 6: show what the deadline can and cannot do. Each attempt below takes 8 seconds and times out, as an 8-second per-attempt timeout would.

Verified: capstone/fde-product-discovery-copilot/lessons/M2-L4/test_m2_l4_resilient.py

def test_the_deadline_stops_new_attempts_but_cannot_interrupt_one_in_flight() -> None:
    t = FakeTime()

    class Slow:  # every attempt burns 8 s (its per-attempt timeout) and then times out
        name = "slow"
        attempts = 0

        def complete(self, messages, schema=None) -> Completion:  # noqa: ANN001
            self.attempts += 1
            t.now += 8.0
            raise ProviderTimeout("8 s")

    slow = Slow()
    with pytest.raises(ProviderTimeout):
        ResilientProvider(slow, clock=t.clock, sleep=t.sleep, jitter=lambda d: d, deadline_s=10.0).complete(MSG)  # type: ignore[arg-type]
    assert slow.attempts == 2 and t.waits == [0.5] and t.now == 16.5  # 8 + 0.5 + 8: past the 10 s deadline

The second attempt starts at 8.5 seconds and ends at 16.5, past the 10-second deadline; the third never starts. With the SDK defaults the same loop could hold a user for minutes. Run all of it:

./.venv/Scripts/python -m pytest lessons/M2-L4 -q

Real output of that run (the timing varies):

..............                                                           [100%]
14 passed in 22.96s

Exercise

Add a retry budget. In a bad minute every one of the nine calls in a request may retry, turning 9 calls into up to 27. Give ResilientProvider an optional RetryBudget, created once per discovery request, that allows at most 3 retries across all calls of that request.

Acceptance criteria: with nine calls that each fail once and then succeed, the fourth failing call raises instead of sleeping; the first three complete; a provider that never fails never touches the budget. Done when the tests pass and the budget object is the only new state.

Hint 1: where does the counter live?

Not inside complete: it runs once per call, so a local counter resets nine times. The state belongs to the request.

Hint 2: where does it plug in?

Create it where run_discovery creates its UsageMeter, share it with every wrapper, and spend one unit just before the sleep. With no units left, re-raise.

Common mistakes

Retrying everything. A 400 or 401 never succeeds on a second try. Retry only the errors in your closed set.
Two retry owners. The SDK's retries under your own loop multiply attempts (3 x 3 = 9) and hide the worst case.
No deadline. "3 attempts with backoff" is bounded in count, not in time; a 10-minute per-attempt timeout makes three attempts half an hour.
Counting requests, not calls. One request here makes nine model calls. Limits, bills and latency are all in calls and tokens.

Knowledge check

1. Why does the pipeline depend on `LLMProvider` and not on the vendor SDK?

A one-method protocol lets you swap, wrap and fake providers without editing the pipeline. A direct SDK import would tie every caller to one vendor and make failure tests depend on the network.

2. The mock returns valid JSON every time. What does that tell you about a real model's reliability?

Nothing. The mock is a rule-based function, so its reliability is a property of the rules. A real model can truncate, wrap JSON in prose or time out, which is why failure tests use scripted providers.

3. A 429 arrives with `retry-after: 30` and your deadline is 10 seconds. What should the wrapper do, and why?

Raise immediately. Retrying sooner would fail, and waiting 30 seconds would break the promise made to the user. A fast, honest error lets the caller show a message or queue the work.

4. Why must `ErrorMappingProvider` sit inside `ResilientProvider`, not outside it?

The retry loop catches only the closed set of ProviderError subclasses, so a raw SDK exception must be translated first, closest to the SDK. A 400 or 401 then propagates untouched and is never retried.

5. Thirty requests a minute looks small against a limit of 200 requests a minute. Why can it still fail?

Each request makes nine model calls, so the provider sees 270. Convert requests into calls and tokens before comparing with a limit.

You can now

Describe the LLMProvider contract and wrap it without changing callers.
Say why the mock is not evidence about a real model.
Write bounded retries with a server-given wait and a deadline, and test them with a fake clock.
Explain the three timeouts and the one-retry-owner rule.
Compute the cost of a request from recorded tokens and a placeholder price.

LLM Provider Calls: Adapter, Mock, Rate Limits, Cost

At a glance

Scenario

What you will be able to do

Prerequisites

Core ideas

One contract for every provider

The mock is a stand-in, not a model

What goes wrong on the way to a real provider

Three timeouts, three owners

Recording usage

Worked example

Build it

Exercise

Common mistakes

Knowledge check

You can now

Further reading

Next

Enroll to keep reading

Ask about this article

Enjoyed this? Get more like it
every Monday.

Comments

At a glance

Scenario

What you will be able to do

Prerequisites

Core ideas

One contract for every provider

The mock is a stand-in, not a model

What goes wrong on the way to a real provider

Three timeouts, three owners

Recording usage

Worked example

Build it

Exercise

Common mistakes

Knowledge check

You can now

Further reading

Next

Enroll to keep reading

Ask about this article

Enjoyed this? Get more like it every Monday.

Comments

Enjoyed this? Get more like it
every Monday.