All posts
    AI
    July 16, 20264 min read

    LLM Token Budgeting: Sizing Prompts, Context, and Cost

    A practical guide to budgeting tokens in LLM apps — how tokens map to cost and context limits, how to estimate prompt size, and the levers that keep spend predictable.

    #llmops#llm#cost#tokens
    Share

    Every LLM call is billed by the token, and every model has a hard context limit measured in tokens. If you don't budget them, two things happen: your costs drift unpredictably, and requests start failing when a stuffed prompt overflows the window. Token budgeting is the discipline of knowing — before you ship — roughly how many tokens each request uses and what that costs at scale. Here's how to do it.

    Tokens 101 (just enough)

    A token is a chunk of text — for English, roughly 4 characters or ¾ of a word on average. Both your input (system prompt + context + user message) and the model's output are counted, and providers usually price them separately (output often costs more than input).

    Two numbers govern every request:

    • Cost = input_tokens × input_price + output_tokens × output_price
    • Fit = input_tokens + max_output_tokens must stay under the model's context window
    TIP

    Estimating a prompt's size shouldn't be guesswork. Paste your system prompt, context, or a RAG chunk into the free LLM Token Counter to see the token estimate and how much of each model's window it uses.

    Where your tokens actually go

    For a typical RAG or agent request, the input is dominated by things you control:

    • System prompt — sent on every call. Trim it; it adds up fast at scale.
    • Retrieved context (RAG) — usually the biggest chunk. topK × chunk size = your context bill.
    • Conversation history — grows every turn if you don't window it.
    • Tool definitions — each tool's schema costs tokens on every call.
    • The user message — usually the smallest part.

    Output tokens are governed by the task and your max_tokens cap.

    The levers that control the budget

    1. Cap output — set max_tokens to what the task actually needs. Uncapped output is the #1 surprise cost.
    2. Right-size RAG — a topK of 4–6 focused chunks beats 20 mediocre ones, and costs a quarter as much. Tune chunking so each chunk earns its tokens.
    3. Window the history — keep the last N turns (or summarize older ones), don't replay the whole conversation.
    4. Trim the system prompt — every wasted sentence is billed on every request. Move rarely-needed instructions into tools or conditionals.
    5. Use prompt caching — for stable prefixes (system prompt, big context), prompt caching can cut input cost dramatically on repeated calls.
    6. Match the tier to the task — high-volume extraction doesn't need a frontier model; a fast/cheap tier slashes the per-token price.

    A simple budgeting method

    1. Estimate one request. Add up system prompt + typical context + history + expected output tokens (use the token counter for the text-heavy parts).
    2. Cost it. Multiply by the model tier's input/output prices — the LLM Cost Estimator does this across models.
    3. Scale it. Multiply by expected requests/day × 30 for a monthly figure.
    4. Add headroom + set caps. Add ~20% buffer, set max_tokens, and alert on anomalies.
    5. Meter in production. Extract the usage token counts from each response and track per feature/consumer — an AI gateway makes this centralized.

    FAQ

    How accurate is the ~4-chars-per-token rule? Good enough for planning English prose. Code, non-English text, and rare words tokenize differently — use it for budgets, not exact billing.

    Input or output — which should I optimize first? Depends on your ratio. RAG/agent apps are usually input-heavy (context dominates) — trim context and cache. Chat/generation apps are output-heavy — cap max_tokens.

    Does a bigger context window mean I should use it? No. Just because a model accepts 1M tokens doesn't mean you should send them — you pay for every one, and more context often dilutes the answer. Send the least context that answers the question.

    How do I stop one team blowing the budget? Meter and quota per consumer at an AI gateway, so each team has an enforceable token budget.

    Token budgeting turns "why is the bill unpredictable?" into a number you set on purpose. Estimate a request, cost it, scale it, cap it, and meter it — and your LLM spend becomes a lever you control instead of a surprise.

    Try it now: estimate your prompt with the LLM Token Counter, then price it with the LLM Cost Estimator. More: prompt caching and LLM observability.

    Need help getting an AI feature's unit economics under control? Let's talk.

    Ask about this article

    Get answers grounded in this post. AI-generated — based on this article, and may be imperfect.

    Was this helpful?
    AY
    Avaneesh Yadav

    I build enterprise AI systems — Spring AI, RAG, and agents — and write about shipping LLMs to production. I also run advisory and workshops for engineering teams.

    Work with me About LinkedIn GitHub
    NewerClaude vs GPT for Enterprise: An Honest ComparisonOlder Modern Java for Backend Developers: The Features That Actually Matter

    Read next

    Article

    LLM Observability in Production: Logs, Traces, and Metrics That Matter

    What to measure, log, and alert on in a production LLM app — latency, token usage, error rates, retrieval quality — and the tools and patterns to get there.

    Article

    Prompt Engineering Best Practices for Enterprise Teams

    Beyond \"be clear and specific\" — the prompt engineering techniques enterprise teams actually ship: system-prompt architecture, few-shot design, retrieval grounding, structured outputs, and iterating with evals.

    Article

    AI Code Review at Scale: How We Use Claude to Review Every PR Before Humans See It

    Six months, 4,200 pull requests, and a GitHub Actions workflow that catches real bugs before your team spends a single minute on review. The architecture, the prompt, the cost, and what we learned.

    Scaled AI Weekly

    Enjoyed this? Get more like it every Monday.

    Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.

    Join engineers building production AI systems

    Free: LLM Production Readiness Checklist (PDF)

    50 checks across observability, rate limiting, cost optimization, failure handling, and security — for teams shipping AI features to production.

    No spam. Unsubscribe any time.

    Comments

    Back to all posts