Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to LLMOps and observability for AI systems.
Context
A telecom operator has three teams on one model-provider account and one API key. All figures use placeholder cost units, never real prices; see the LLM cost estimator for your own rates.
| Team | Feature | Requests a month |
|---|---|---|
| Customer care | Chat assistant | 2,000,000 |
| Network operations | Incident summariser | 150,000 |
| Billing analytics | Nightly bill explanations | 600,000 |
Problem statement
The month-end invoice was 10,440 units. Finance split it equally, 3,480 units to each team, and each lead disputed the share. Nobody could say which feature or prompt version drove the total.
- Target 1: at least 95% of spend attributed to a team, feature and version.
- Target 2: an alert at 80% of each team's budget, before month end.
- Target 3: a weekly report the three leads read.
Constraints
- Fixed: the three teams keep shipping prompt changes on their own schedules.
- Fixed: one provider invoice, in units per million tokens of input and output.
- Limit: attribution may not add a visible delay to a request.
- Limit: a team must not be able to relabel its spend as someone else's.
Why the traditional approach fell short
An equal or headcount split is fair to no one: here it charged network operations 3,480 units for 2,280 of real use. A provider invoice shows totals per account, not per feature or prompt version. Cloud-resource tags cover servers, not tokens, so model spend falls through.
Options considered
| Option | Verdict | Reason |
|---|---|---|
| A. Split the invoice by headcount or equally (non-AI, finance only) | Rejected | Free, but wrong by up to 1,920 units for one team, and it explains nothing |
| B. One provider account or key per team | Kept as a guard | Gives team totals from the invoice; no feature or version, key sprawl, no check before month end |
| C. Tags at the Model gateway, a usage ledger, budgets, a weekly report | Accepted | Attributes by team, feature and version, and enforces budgets while the month is running |
Solution architecture
Figure 1. Tags and the ledger at the gateway
How to read this: every call passes the gateway, so the ledger is complete for gateway traffic; the dotted line is the invoice the ledger must match.
Figure 2. One tagged call
How to read this: the alt block shows that budget enforcement acts on non-urgent work only, so a team's chat never stops because a nightly job overspent.
How AI solved it, step by step
Step through one call, then compare with the old month.
Under the hood
A ledger record is tokens times a rate. With placeholder rates of 1 unit per million input tokens and 4 per million output tokens, one care request is (1,500 x 1 + 300 x 4) / 1,000,000 = 0.0027 units.
The tag set decides what you can later ask.
| Tag | Source | Why |
|---|---|---|
| team | Credential | Cannot be relabelled by a caller |
| feature | Caller, allow-listed | Splits one team's spend |
| version | Caller, allow-listed | Ties a cost change to a prompt release |
| environment | Credential | Keeps test traffic out of production totals |
Keep the values few, so the number of combinations stays small; put per-user detail on the trace, not on the cost tag. What a tag cannot attribute: calls that bypass the gateway, fixed costs such as hosting an index, and the effort of human review. The mechanism is in Under the hood: Cost attribution tags.
What went wrong / failure modes
| Failure | Mitigation |
|---|---|
| Traffic bypasses the gateway (here a prototype script with the raw key: 300 units) | Weekly reconciliation against the invoice; then rotate the key so only the gateway holds it |
| A caller mislabels feature or version | Allow-list values; derive team from the credential |
| Tag explosion from free text | Fixed vocabulary; free text goes on the trace |
| A budget stops customer chat | Enforce only on non-urgent calls; urgent calls alert instead |
| Rate card out of date | Store the rate-card version on each record and recompute when rates change |
Results
All numbers are illustrative targets, not measurements.
| Measure | Before | After (target) |
|---|---|---|
| Spend attributed to team, feature and version | 0% | 97.1% (10,140 of 10,440) |
| Care share, from ledger | Unknown (billed 3,480) | 5,400 units (51.7%) |
| Network operations share | Unknown (billed 3,480) | 2,280 units (21.8%) |
| Billing analytics share | Unknown (billed 3,480) | 2,460 units (23.6%), 123% of a 2,000 budget |
| Budget alert | None | At 80%, about day 20 for billing |
Tradeoffs and when NOT to use this approach
- One small team. A single team with one feature needs a monthly look at the invoice, not a ledger.
- Gateway as a dependency. Every call crosses it, so it needs the resilience of any shared service.
- Attribution is not accuracy. A tag says who spent, not whether the spend was worth it; pair it with a quality metric.
- Chargeback politics. Showback (report only) is a safer first step than billing teams.
Design rules extracted
- Tag at the gateway and derive the team from the credential; trust no caller-supplied team.
- Put version in the tag, so a cost change maps to a release.
- Reconcile the ledger with the invoice every week; the gap finds what bypasses you.
- Enforce budgets on non-urgent work only; alert for the rest.
- Keep cost tags few; keep detail on the trace.
Assumptions
All numbers are illustrative; prices are placeholder units per million tokens, input 1 and output 4, never real rates.
- Care: 2,000,000 requests x 1,500 input tokens = 3,000 million input tokens = 3,000 units; 2,000,000 x 300 output = 600 million x 4 = 2,400 units; total 5,400.
- Network operations: 150,000 x 12,000 = 1,800 million input = 1,800 units; 150,000 x 800 = 120 million x 4 = 480 units; total 2,280.
- Billing analytics: 600,000 x 2,500 = 1,500 million input = 1,500 units; 600,000 x 400 = 240 million x 4 = 960 units; total 2,460.
- Gateway ledger total 5,400 + 2,280 + 2,460 = 10,140. Invoice 10,440. Gap 10,440 - 10,140 = 300 units (2.9%); attributed 10,140 / 10,440 = 97.1%.
- Equal split 10,440 / 3 = 3,480 each. Shares of the invoice: 5,400 / 10,440 = 51.7%, 2,280 / 10,440 = 21.8%, 2,460 / 10,440 = 23.6%, gap 2.9%.
- Budgets (invented): care 5,500 (98% used), network operations 2,500 (91%), billing 2,000 (123%). Billing daily rate 2,460 / 30 = 82 units; 80% of budget = 1,600 units, reached on day 19.5, so the alert fires about day 20.
- One care request: (1,500 x 1 + 300 x 4) / 1,000,000 = 0.0027 units; 0.0027 x 2,000,000 = 5,400, matching the team total.
Last verified: 2026-10-22