LLM Observability in Production: Logs, Traces, and Metrics That Matter

• Refusals ("I can't help with that") — the model responded but produced no useful output.
• Truncation — response hit max_tokens and was cut off.
• JSON parse failures — structured-output requests that returned malformed JSON.

Detect these with a post-response check and log them as a separate metric. A rising refusal rate often signals a prompt that's drifting toward policy boundaries.

4. Retrieval quality (for RAG)

• Retrieval latency — vector search is usually fast (< 100ms), but embedding calls are not.
• Chunk count returned — are you consistently getting your target topK?
• Similarity scores — if your vector store returns them, log the top-1 score. A low score means the query hit your corpus but nothing was actually relevant.
• Citation accuracy (offline) — periodically sample answers and check whether cited documents actually support the answer.

A structured log pattern for LLM calls

Emit one structured log event per LLM call with everything you need to debug later:

{
  "event": "llm_call",
  "trace_id": "abc123",
  "span_id": "def456",
  "feature": "ask_article",
  "model": "llama-3.1-8b-instant",
  "latency_ms": 1842,
  "input_tokens": 3102,
  "output_tokens": 248,
  "cost_usd": 0.00047,
  "finish_reason": "stop",
  "retrieval_chunks": 4,
  "retrieval_top_score": 0.87,
  "error": null
}

Key principles:

• Never log the prompt or completion — they will contain user data and PII. Log metadata, not content.
• Always log finish_reason — stop (clean), length (truncated), content_filter (refused).
• Attach a trace_id so you can join LLM calls to the user request they belong to.

Tracing across services

For multi-step pipelines (retrieve → rerank → generate), use distributed tracing. Each step gets a span under the same trace ID:

trace_id: abc123
  span: retrieve        latency=45ms  chunks=4  top_score=0.87
  span: rerank          latency=12ms  chunks=3
  span: llm_generate    latency=1842ms  tokens=3350  cost=$0.00047

OpenTelemetry is the standard instrumentation layer. Langfuse is purpose-built for LLM tracing and integrates directly with most frameworks (including Spring AI).

Alerting thresholds to start with

Start conservative — too many alerts train people to ignore them.

Tool landscape

• Langfuse — purpose-built LLM observability; traces, evals, cost tracking, prompt versioning. Free tier is generous; self-hostable.
• OpenTelemetry — vendor-neutral instrumentation for spans and metrics; exports to your existing stack (Prometheus, Jaeger, Grafana).
• Prometheus + Grafana — standard metrics/dashboards; add your LLM metrics via custom counters.
• AI gateway (Apigee, Kong AI, LiteLLM) — centralized token metering and cost allocation without instrumenting every service.

FAQ

Do I need all of this from day one? No. Start with structured logging (the JSON pattern above) and token cost tracking — those two give you the most signal for the least work. Add tracing and eval when you're debugging retrieval or multi-step failures.

How do I avoid logging PII? Log metadata (token counts, scores, latency, IDs) never prompt content or completions. For compliance, make this a code standard, not a documentation note.

What's the difference between monitoring and evaluation? Monitoring is real-time signal (is it up, is it fast, is it spending what it should?). Evaluation is offline quality assessment (are the answers good?). You need both — see testing AI applications.

My LLM app is simple — do I still need this? At minimum: log token usage per request and track the error rate. Everything else scales with complexity.

LLM observability is the difference between flying blind and knowing when your AI degrades. Instrument latency, tokens, errors, and retrieval quality; emit structured logs without PII; trace multi-step pipelines; and alert on the signals that actually predict user experience.

More: the LLM Observability Guide, AI gateway for centralized metering, and token budgeting.

Shipping an LLM app to production and want help with your observability stack? Let's talk.