LLMOps and Observability for AI Systems

Privacy of traces

Figure 5. Where a trace may hold sensitive text

How to read this: the solid path is what every engineer can open, the dotted path is optional and guarded, and nothing carrying raw text leaves the boundary by default.

The OpenTelemetry GenAI conventions mark the message-content attributes as opt-in. They warn that the input and output messages are "likely to contain sensitive information including user/PII data" (OpenTelemetry GenAI conventions, client inference, as of 2026-10-22). Page 12 covers redaction methods and their limits.

Privacy also constrains debugging. In its September 2025 postmortem Anthropic says its privacy and security controls limit when engineers can view user interactions, which "protects user privacy but prevents engineers from examining the problematic interactions" needed to reproduce bugs (Anthropic Engineering, 17 September 2025). Design the trace so identifiers and structure answer most questions without raw text.

Case files

• Canary release of a prompt change. In a composite SaaS scenario, a prompt change went to everyone and quality fell for one customer segment. Pattern: version pin, shadow then canary with a confidence-interval gate, and a one-click rollback.
• Explaining a shared model bill. In a composite telecom scenario, three teams shared one model account and the monthly bill could not be explained. Pattern: tags at the gateway, per-team budgets and a weekly report.
• Anthropic's September 2025 quality postmortem. A public case based on Anthropic's engineering post about three infrastructure bugs that degraded Claude responses, which users reported before the causes were found. Pattern: the evaluation, deployment-test and privacy-safe debugging changes the post lists.

Under the hood: The trace and span data model

Intuition

A trace is a family tree of timed operations for one request. Every span records its parent, so the tree can be rebuilt after the fact. The only things that must cross a network call to make this work are two IDs.

Mechanism

Figure 6. Context propagation across one AI request, with a provider timeout

How to read this: the header travels with the request while the spans travel separately to the trace store, so a slow or down store cannot slow the answer.

1. The first service creates the root span and a trace ID.
2. Before every outgoing call it writes the IDs into a header. OpenTelemetry's default propagator uses the W3C traceparent header (OpenTelemetry, Context propagation).
3. The receiver reads the header and creates its own span in the same trace, with the caller's span as parent.
4. Spans are exported separately. The backend joins them by trace ID.
5. For asynchronous work, a producer span creates a job and a consumer span processes it, possibly long after. Span links associate spans across traces (OpenTelemetry, Traces).

The W3C Trace Context Recommendation (23 November 2021) fixes the header format (W3C Trace Context).

An all-zero trace-id or parent-id is invalid. The companion tracestate header carries vendor-specific information and allows at most 32 list members.

The GenAI semantic conventions. As of 2026-10-22 OpenTelemetry's generative-AI conventions carry the status "Development". The pages on opentelemetry.io are notices that say the content moved to the open-telemetry/semantic-conventions-genai repository. OpenTelemetry's document-status page says the support guarantees and allowed changes depend on a document's maturity level, so plan for names and requirement levels to change.

The inference span name should be {gen_ai.operation.name} {gen_ai.request.model} and its kind should be CLIENT, with INTERNAL allowed for in-process models.

What this means for architects

Under the hood: Prompt registry and versioning

Intuition

A release is the whole recipe for an answer: prompt, model, settings, tools and retrieval. If any piece can change without a new version, nobody can say what produced a bad answer or what "roll back" restores.

Mechanism

Figure 7. What one release record pins

How to read this: the release is a bundle of immutable parts, and the pointer is the only mutable thing, which is why a rollback is a pointer move.

1. Assemble the release from immutable parts and give it an ID.
2. Run the gate against that ID and store the outcome with it.
3. A pointer (an alias or a flag) says which release serves each traffic slice.
4. Every span carries the release ID, so a bad answer names its release.
5. Rolling back moves the pointer to the previous release. Its parts have not changed.

MLflow's prompt registry documents the same split: prompt versions "are immutable", each has a sequential number, and an alias is a "mutable named reference to the prompt" that teams can use for the production version and for roll-backs. Its model_config is "mutable", unlike the prompt template (MLflow, Prompt Registry, as of 2026-10-22). So a registry entry alone is not a release record: pin the settings in the release too.

Anthropic's models page says every Claude model ID is a pinned snapshot and lists a "not sooner than" retirement date per current model for the platforms Anthropic operates (Anthropic, Models overview, as of 2026-10-22). Amazon Bedrock and Google Cloud set their own dates, and other providers differ, so check each one.

What this means for architects

Under the hood: Shadow, canary and A/B releases

Intuition

A shadow is a rehearsal, a canary is a small real exposure, and an A/B test is a randomised experiment. Google's SRE Workbook defines canarying as "a partial and time-limited deployment of a change in a service and its evaluation" (Google SRE Workbook, Canarying Releases).

Mechanism

Figure 8. Shadow, then canary, with a gate and a rollback

How to read this: in shadow nobody sees the candidate, in canary a few do, and the same controller reads the same evidence and picks one of three outcomes.

1. Shadow. The gateway copies requests to the candidate. Istio documents the same mechanism: mirrored traffic happens "out of band of the critical request path" and is "fire and forget", so responses are discarded (Istio, Mirroring).
2. Canary. A small share of users get the candidate's answer. The Workbook says a canary "should be sizeable and last long enough to be representative of the overall deployment", and that its authors "strongly advise running only one canary deployment at a time".
3. Judge a sample. Judging every canary answer is rarely worth the spend, so a share is judged (35% here, an assumption).
4. Decide with an interval. Compute the Wilson interval from page 8 and apply the three-outcome rule.
5. A/B. Assign users randomly and consistently to a variant, then compare a user-outcome metric with a sample size chosen in advance.

The exposure arithmetic uses 20,000 requests per day, a judged share of 35% and a drop from 92% to 88% that needs 349 items to detect (computed in the worked example).

The Workbook's own example shows the principle: a release that fails 20% of requests, on a 5% canary, produces a 1% overall error rate.

The starting interval is 89.2% to 94.1%, so the gate says fail. Set passes to 324, then to 305 and 304, and watch the bounds move.

The starting state, 321 of 349 (92.0%), matches the baseline and still does not clear the bar. This is the proof-gate behaviour that page 8 describes. For early stages, use the regression rule instead: block when passes fall to 312 or fewer of 349. This is the one-sided test at n = 349 whose exact binomial size and power page 8 computes: it blocks 5.0% of the time at a true 92% and detects a true 88% about 81% of the time.

One caution comes from a simulation made for this page, not a sourced method. Testing against 92% after every 50 items up to 350, and blocking if any look fell below the one-sided 5% line, raised the false-block rate at a true 92% from about 5% to about 15% (20,000 simulated runs). Fix the sample size or plan the looks.

What this means for architects

Under the hood: Cost attribution tags

Intuition

A provider bill has one line per model. The business asks per team and per feature. A tag stamped on every call at the gateway is what turns tokens into "who and why".

Mechanism

Figure 9. Tags stamped at the gateway and reconciled against the provider

How to read this: the left path stamps and records, the right path is the provider's own number, and the difference between them is the spend your tags cannot explain.

1. Derive tags from authenticated identity and the route, not from text a caller supplies. This is the course's rule.
2. Record per call: tags, model, uncached input, cached input and output tokens, outcome and retry count. Record tokens, and apply a price table later. This page states no prices.
3. Roll up daily by tag.
4. Reconcile with the provider. Anthropic's Usage API groups usage by dimensions such as model, workspace, API key and service tier, and its Cost API groups costs by workspace or description (Anthropic, Usage and Cost API, as of 2026-10-22). The page lists no per-request custom tag, so a team-and-feature split comes from your gateway or from mapping keys and workspaces to teams.
5. Report weekly and compare with each team's budget. A gateway can do the tagging: LiteLLM's proxy documentation describes spend tracking for keys, users, teams and tags (LiteLLM, Spend tracking, as of 2026-10-22).

What this means for architects

Decision framework

Figure 10. Choosing the next step for a proposed change

How to read this: only one question lets a change skip the ladder, and every other path ends in one of three named outcomes.

Comparison of the main options

Failure scenarios to design for

Worked example

Release rel-0143 changes only the system prompt. The current release rel-0142 passes 92% on the rubric, and the requirement is a pass rate of at least 90%.

Assumptions:

• Traffic is 20,000 requests per day, 5% go to the canary, and 35% of canary answers are judged.
• The change is worth catching if the pass rate falls from 92% to 88%.
• One-sided alpha is 0.05 and power is 0.80 (z = 1.645 and 0.8416).
• All numbers are illustrative.

The block rule in step 8 is the regression rule above. The 95% one-sided interval for 307 of 349 is about 84.8% to 90.5%: it straddles the 90% bar, so this is evidence of a drop from the baseline, not proof of falling below the bar.

Change p1 to 0.90 and the items needed rise to about 1,221 (1,271 with the correction), because halving the drop roughly quadruples the sample.

Code sample: tracing a model call and passing the context

This sample uses opentelemetry-sdk (1.45.1 was the latest release on PyPI on 2026-10-22). It records the GenAI attributes (status Development) beside our own app.* attributes, marks an error properly, and writes a traceparent header for the next hop.

from opentelemetry import trace
from opentelemetry.propagate import extract, inject
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
from opentelemetry.trace import SpanKind, Status, StatusCode

provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("support-assistant")


def call_model(release_id: str, model_id: str, question: str) -> str:
    attrs = {
        "gen_ai.operation.name": "chat",
        "gen_ai.provider.name": "example-provider",
        "gen_ai.request.model": model_id,
        "app.release.id": release_id,  # our own namespace, not OpenTelemetry's
        "app.feature": "returns-help",
    }
    with tracer.start_as_current_span(f"chat {model_id}", kind=SpanKind.CLIENT, attributes=attrs) as span:
        try:
            span.set_attribute("gen_ai.usage.input_tokens", 812)  # from the provider response
            span.set_attribute("gen_ai.usage.output_tokens", 164)
            return "answer"
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR))
            span.set_attribute("error.type", type(exc).__name__)
            raise


with tracer.start_as_current_span("handle_question", kind=SpanKind.SERVER):
    call_model("rel-0142", "model-x", "What is the returns window?")
    headers: dict[str, str] = {}
    inject(headers)  # writes a traceparent header for the next hop

with tracer.start_as_current_span("worker", context=extract(headers), kind=SpanKind.CONSUMER):
    pass  # same trace ID; its parent is the span that injected the header

The token counts are fixed numbers here, standing in for the provider response. Message content is deliberately not recorded. In production, read the counts from the response and add the content attributes only behind an opt-in.

Common mistakes

• Versioning the prompt but not the settings or retrieval config. Pin the whole release record, not the text.
• Logging 200 and calling it observed. Record release ID, passage IDs, token counts and finish reason on spans.
• Judging the canary on too few items. Compute the items needed first and use small stages for hard signals only.
• Checking the gate until it turns green. Fix n, or plan the looks, and treat "hold" as a result.
• Letting teams label their own spend. Stamp tags from identity at the gateway.
• Storing raw prompts in the trace store by default. Keep content opt-in, redacted and short-lived.

Try it

Time: 30 minutes. Run the code sample with the console exporter. Add a second function that stands in for your retriever and records passage IDs as an attribute. Write your own release record as a small file: prompt hash, model ID and retirement date, parameters, tool schema hash and retrieval settings. Then compute how many judged items you need to detect a 4-point drop from your own baseline and how many days your canary share takes to produce them.

Done when one trace ID appears on spans from two functions, a release record exists for your current setup, and you can state the smallest drop your canary can detect in one day.

Key takeaways

• A trace is a tree of spans joined by a trace ID that travels in the traceparent header.
• The GenAI semantic conventions are in Development as of 2026-10-22: isolate them behind one module.
• A release pins prompt, model, parameters, tools and retrieval, and a rollback is a pointer move.
• Shadow shows real inputs without exposing users, canary exposes few, and A/B measures user outcomes.
• A gate has three outcomes: promote, hold or block. The number of judged items sets the smallest detectable drop.
• Stamp cost tags at the gateway from identity and reconcile with the provider to find the untagged share.
• Traces hold personal data unless you design them not to.

Self-check

Further reading

• OpenTelemetry: Traces and Context propagation.
• W3C Trace Context Recommendation (23 November 2021).
• Google SRE Workbook, chapter 16: Canarying Releases.
• OpenTelemetry GenAI semantic conventions (status Development as of 2026-10-22).
• Anthropic Engineering: A postmortem of three recent issues (17 September 2025).

Last verified: 2026-10-22

Next

Page 10: Reliability and scaling for AI systems