CCA Domain 5: Context Management & Reliability

CCA Foundations course · Page 6 of 8 · ← Back to all courses · Weight: 15%. Mark complete at the bottom to advance.

5.1 — Context Window Management · Core

• The API is stateless — every request carries the full conversation; dropping history (e.g. handing a fresh agent only the latest message) loses established facts and forces the customer to repeat themselves. This is the same statelessness from Domain 1.1, applied here to why context management is a design problem rather than something the model handles for you.
• Case-facts block at the TOP, verbatim — an authoritative structured layer (order ID, amounts, dates, status) included in every prompt, outside the summarised history, so it survives /compact and progressive summarisation instead of being paraphrased into vague prose. This is an addition to full history, not a replacement for it.
• Lost-in-the-middle effect: models process the beginning and end of long inputs reliably but can omit findings buried in the middle — put a key-findings summary at the start of aggregated input, with explicit section headers. This is a positional effect, not a relevance judgment — a genuinely critical fact buried at position 40 of 60 is still at risk purely because of where it sits.
• Trim verbose tool output to only relevant fields (a 40-field order lookup down to the 5 that matter) before it accumulates in context, not after — trimming retroactively via /compact still means the model reasoned over 35 irrelevant fields for every turn in between.
• Multi-agent handoffs: require subagents to return structured data (key facts, citations, relevance scores, source dates) instead of verbose content and full reasoning chains — especially when the downstream agent has a limited context budget.
• Full history is a floor, not a ceiling. The persistent case-facts block sits alongside the complete conversation history, not in place of it — dropping full history to save space (handing a fresh agent only "the latest message plus the facts block") still loses the narrative context a customer expects the agent to remember, even with the hard numbers intact.

5.2 — Escalation Patterns · Core

• Explicit human request = immediate escalation, no exceptions — honour it before attempting any investigation.
• Policy gap / silence on the request = escalate (e.g. competitor price-matching when policy only covers own-site adjustments).
• Sentiment-based or self-reported-confidence escalation = anti-pattern — sentiment doesn't correlate with case complexity, and self-reported confidence is poorly calibrated (the agent is often confidently wrong on the hardest cases).
• Ambiguous matches: when a tool returns multiple results, ask for an additional identifier rather than picking one heuristically.
• Fix a low autonomous-resolution rate with explicit escalate-vs-resolve criteria + few-shot examples, not a confidence threshold — a confidence-based fix targets the wrong signal entirely.
• The two failure directions are symmetric and both costly. An agent that escalates too readily wastes human capacity on cases it could have resolved; one that resolves too readily mishandles policy-gap cases it should have handed off. Explicit criteria fix both directions at once — a confidence threshold only ever tunes one dial and still misses policy-gap cases entirely, since those are a category problem, not a confidence problem.

5.3 — Error Propagation · Core

• Silent suppression AND full abort are both wrong. Marking a failed subagent's empty output as success hides the error; terminating the whole workflow on one subagent failure throws away recoverable work.
• Return a structured error with failure_type, attempted_query, partial_results, and alternatives — preserve successful work and annotate the gaps so the coordinator can decide: retry with a modified query, try another approach, or proceed with what's there.
• Local recovery first — subagents retry transient failures with exponential backoff a few times locally, and only propagate what they genuinely can't resolve. Propagating every transient hiccup up to the coordinator turns a self-healing system into one that pauses on every network blip.
• Access failure ≠ valid empty result — a timeout needs a retry decision; a successful query that found nothing is a legitimate result. Conflating the two hides information the coordinator needs.
• Coverage annotations on synthesis output — WELL_SUPPORTED (corroborated), PARTIAL (some support / incomplete), COVERAGE_GAP (a topic area left unaddressed because sources were unavailable) — tell the coordinator how much to trust each finding.
• Structured error context is what makes retry-vs-escalate a coordinator decision instead of a guess. Without attempted_query and alternatives, the coordinator can only retry blindly (same query, same failure) or give up — neither is actually a decision, just a coin flip dressed up as one.

5.4 — Context Management in Large Codebase Exploration · Core

• Context degrades in extended exploration — the model starts giving inconsistent answers and referencing "typical patterns" instead of the specific classes it actually discovered earlier. The fix is never "just keep going."
• Scratchpad files — have agents record key findings to a file (.scratchpad.md) and reference it for later questions, counteracting degradation as the session grows.
• Subagent delegation for specific focused questions ("find all test files," "trace the refund-flow dependencies") keeps the main agent's own context small and focused on coordination.
• Summarise between phases — condense one exploration phase's key findings before spawning subagents for the next, injecting the summary into their initial context instead of their inheriting nothing or everything.
• Structured state for crash recovery — each agent exports state to a known manifest location; on resume, the coordinator reloads the manifest and injects it into agent prompts.
• /compact compresses conversation history to reclaim context — use it alongside scratchpad files and manifest exports, which persist the specific findings /compact would otherwise blur, so you get room back without losing key facts.
• Recognise the same degradation signature in different contexts. "Referencing typical patterns instead of the specific classes found" during codebase exploration is the same underlying failure as a support conversation losing an exact refund amount (5.1) — long-running context slowly substituting vague generalities for concrete facts. The fix in both cases is a persistent, structured record kept outside the part of the context that gets compressed.

5.5 — Human Review & Confidence Calibration · Core

• Aggregate accuracy hides per-segment failure — a "97% overall" figure can mask a specific document type or field performing badly. Validate accuracy by document type and field segment, not just in aggregate, before reducing human review.
• Field-level confidence scores, calibrated against a labeled validation set (e.g. via ECE — expected calibration error), determine where review attention goes — raw model confidence ≠ real accuracy until it's been calibrated this way. A model reporting "90% confident" that is only right 70% of the time at that self-reported level is miscalibrated regardless of how good it sounds; ECE quantifies exactly that gap.
• Route to human review: low-confidence extractions and ambiguous/contradictory source documents — prioritising limited reviewer capacity toward where it actually changes outcomes, rather than spreading review thin across everything at a fixed sampling rate.
• Stratified random sampling of high-confidence, auto-accepted extractions keeps measuring the real error rate and catches novel failure patterns that a one-time validation pass would miss.
• Blocking review for irreversible actions; async review for reversible ones — the review mode should match how costly a mistake would be to undo. A wire transfer waits for a human before it executes; a draft categorisation that's trivially correctable later can ship now and be spot-checked after the fact.

⚠ Often-missed — Confidence Scores Are Not Accuracy · Gap

A high aggregate accuracy number is a distractor for "we can cut human review now" — segment-level failure and uncalibrated confidence both hide behind it. Never route on raw model confidence; calibrate first against labelled data, then route on the calibrated probability.

⚠ Often-missed — Escalation vs. Error Propagation Are Different Layers · Gap

Don't conflate 5.2's escalation triggers (when a customer-facing agent hands a case to a human) with 5.3's error propagation (when a subagent reports failure to its coordinator). Both use the word "escalate" loosely in casual conversation, but the exam scenario tells you which layer is in play by who the recipient is — a human reviewer vs. a coordinator process — and the correct fix lives in the matching subsection.