How an LLM Application Works: One Request in Code

What you'll learn: what actually happens when your code "calls an LLM": two messages go out, text and token counts come back, and everything else (checking the reply, filtering what the user may see, deciding whether to call a model at all) is ordinary code that you write. You trace one request, count the calls in the capstone, price them with placeholder prices and write a lookup that needs no model.

At a glance

Figure 1. The capstone application, end to end

How to read this: the model sits behind one interface at the edge of the system; every other box is ordinary code that your tests can run without it.

• One model call is messages in, text plus token counts out. Nothing more.
• The model proposes; code validates, filters and decides what the user sees.
• The pumps request in the capstone makes 9 model calls, not 1, and the bill grows with the calls.
• If the answer sits in a record, a lookup is cheaper, faster and identical every time.
• Input and output tokens have separate prices: you pay for each.

Scenario

Composite scenario, not a real company. Lumen Industrial Supply, a fictional distributor, is piloting a copilot for its sales engineers.

Two kinds of questions arrive: "What is the ingress protection rating of the LUM-VS-100?", which has one answer in the product record, and "monitor vibration on 40 pumps in a humid plant, budget around 30k, must work with our existing OPC UA setup", which needs search, checks and judgment. The lead asks: does every question need a model, and what does one request cost? All figures are illustrative.

What you will be able to do

• Trace one LLM request in code: the messages built, the call made, the reply validated, the tokens recorded.
• Say which steps of the capstone belong to the model and which to code.
• Compute the cost of a request from token counts and placeholder prices, and count the calls a workflow makes.
• Decide when a lookup should replace a model call, and write one.

Prerequisites

You should be able to read Python and run pytest: see Testing with pytest. For the theory of gateways, routing, tokens and context windows, read Models and routing; this lesson adds only the view from the code.

Core ideas

One request, in code

Predict first. The capstone's extract_messages builds the first call for the pumps request. How many messages does it send, and how many model calls does the whole request make end to end (extract, plan, compose)?

Figure 2. Anatomy of one request

How to read this: the model is one participant out of five, and four of them are code you own and can test.

The prompt builder is a plain function. The instructions go in the system message; the customer's text goes in the user message, untouched.

Verified: capstone/fde-product-discovery-copilot/app/llm/prompts.py

def extract_messages(request: str) -> list[Message]:
    system = (
        f"TASK: {TASK_EXTRACT}\n"
        "Turn a customer request for industrial equipment into JSON requirements. "
        "Use only what the request says; leave a field empty when it is not mentioned. "
        "Return JSON only. Never invent a product."
    )
    return [Message("system", system), Message("user", request)]

A real provider's HTTP API has the same shape. As of 2026-11-02 the Anthropic Messages API requires a model id, a list of messages and max_tokens (a ceiling on generated tokens, not a target), and takes the system prompt in a separate system parameter. Its example responses carry a usage object with input and output token counts.

What the model owns and what code owns

The model handles language at the two ends; code handles everything that must be exact, allowed or auditable. The mock is a rule-based stand-in that makes the course runnable offline; a real model behaves less predictably.

When deterministic code beats a model

Decision exercise. Many Lumen questions look like "What is the X of LUM-...?". Options: (A) send every question through the full pipeline; (B) answer single-product field lookups from the record and send the rest to the model; (C) replace the pipeline with keyword rules. Choose one and justify it in three sentences.

At the customer: a sponsor asks "is this AI?" about a lookup. A good answer: the model interprets language, a lookup returns a stored fact, and mixing them gives reliability and a predictable bill.

Cost: tokens in, tokens out, calls

A provider bills input tokens and output tokens at two different prices, which you read from its published price list. One call costs input tokens times the input price plus output tokens times the output price; a workflow multiplies that by its number of calls.

The capstone's UsageMeter prices each call with Settings.input_price_per_mtok and output_price_per_mtok. Both are placeholders (1.0 and 5.0 EUR per million tokens): fill in your provider's published prices before you quote a number.

The start values give 0.004272 units per request and 4.272 per 1,000 requests. Change priceOut to 1 to see how much of the bill output carries; cachedShare shows what caching would do (the capstone does not use it).

Worked example

Assumptions: the mock provider (token counts are the capstone's estimate of 1 word = 1.3 tokens, rounded up, not a real tokenizer); placeholder prices of 1.0 and 5.0 EUR per million input and output tokens; the pumps request as a public user. Numbers come from running the code in lessons/M2-L1.

Cost: (2,792 x 1.0 + 296 x 5.0) / 1,000,000 = 4,272 / 1,000,000 = 0.004272 EUR, so 4.27 EUR per 1,000 requests. Output tokens are 1,480 of 4,272 cost units, 34.6 percent of the bill, although they are only 9.6 percent of the tokens.

Of the 36 golden questions, 3 (q13, q14 and q17) are single-product field lookups that answer_without_model answers with zero calls, 8.3 percent of the set. The same author wrote the set and tuned the pipeline, so treat 3 of 36 as an illustration, not your customer's traffic mix.

Build it

The lesson code is lessons/M2-L1/m2_l1_anatomy.py, with tests in test_m2_l1_anatomy.py. First, the trace: build the messages, call through the metered provider, validate, read the meter.

Verified: capstone/fde-product-discovery-copilot/lessons/M2-L1/m2_l1_anatomy.py

def trace_one_request(text: str, provider: LLMProvider, settings: Settings) -> RequestTrace:
    """Build the messages, make the call, validate the reply, and read the meter."""
    meter = UsageMeter(settings)
    llm = MeteredProvider(provider, meter)
    messages = extract_messages(text)
    requirements = complete_structured(llm, messages, Requirements)
    usage = meter.snapshot()
    return RequestTrace(
        roles=[m.role for m in messages],
        input_tokens=usage.input_tokens,
        output_tokens=usage.output_tokens,
        llm_calls=usage.llm_calls,
        cost_eur=usage.estimated_cost_eur,
        requirements=requirements,
    )

Second, the lookup. It returns None when it cannot answer, which means "send it to the model". It resolves the SKU with catalog.mentions, which ignores products the caller may not see, so a public user asking about a partner-only SKU falls through to the normal path.

Verified: capstone/fde-product-discovery-copilot/lessons/M2-L1/m2_l1_anatomy.py

def answer_without_model(text: str, catalog: Catalog, visibilities: Collection[str]) -> str | None:
    """Answer 'what is <field> of <one SKU>' from the product record. None means: use the model."""
    low = text.lower()
    skus = catalog.mentions(text, visibilities)
    fields = {field for word, field in FIELD_WORDS.items() if word in low}
    if len(skus) != 1 or len(fields) != 1:
        return None
    product = catalog.products[skus[0]]
    field = fields.pop()
    value = getattr(product, field, None) if field in ("ip_rating", "protocols") else product.specs.get(field)
    if value in (None, []):
        return None
    shown = ", ".join(value) if isinstance(value, list) else value
    return f"{product.sku}: {field} = {shown}"

Third, a test that counts calls per task, so an added call fails loudly instead of showing up on the invoice.

Verified: capstone/fde-product-discovery-copilot/lessons/M2-L1/test_m2_l1_anatomy.py

def test_the_whole_pumps_request_makes_nine_calls(services: Services) -> None:
    counting = CountingProvider(MockProvider())
    svc = Services(services.settings, services.catalog, services.store, counting)
    usage = run_discovery(PUMPS, PUBLIC, svc).response.usage
    assert counting.by_task == {"extract_requirements": 1, "plan_next_step": 7, "compose_answer": 1}
    assert (usage.llm_calls, usage.input_tokens, usage.output_tokens) == (9, 2792, 296)
    assert usage.estimated_cost_eur == pytest.approx((2792 * 1.0 + 296 * 5.0) / 1_000_000)

Run the lesson tests:

./.venv/Scripts/python -m pytest lessons/M2-L1 -q

Exercise

Extend answer_without_model so that "What is the price of the LUM-VS-100?" is answered from the record, and the answer says the price is illustrative.

Acceptance criteria:

• A new test shows the price question returns an answer that contains the word "illustrative".
• The golden-question test still finds exactly q13, q14 and q17, or you update it and explain why in a comment.
• q33 ("What is the partner price for the LUM-VS-900?") still returns None for a public user.

Done when the lesson tests pass and you can say why q33 falls through.

Common mistakes

• Treating the model as the application: no validation, no filtering, no limit on calls. Its output is a proposal.
• Counting one call per user request. A tool loop makes many, and each repeats the instructions and history as input.
• Reading only one price. Input and output tokens are priced separately.
• Using a model for a stored fact. It adds cost and a chance of a wrong or stale answer for no gain.

Knowledge check

You can now

• Trace a request from customer text to validated output and name each participant.
• Say which steps of the capstone are model work and which are code work.
• Price a request from token counts and placeholder prices, and count calls with a test.
• Write a lookup that falls through to the model when it cannot answer.

Further reading

• Anthropic Messages API reference: required request parameters, the system prompt and max_tokens.
• Anthropic token counting: why token counts matter for cost, rate limits and routing.
• Models and routing: the architecture view of gateways, routing and context windows.

Last verified: 2026-11-02

Next

Prompts and untrusted input