Domain 7 — Troubleshooting and Optimization
CCA Associate Foundations course · Page 8 of 10 · ← Back to all courses · Weight: 10%. Mark this page complete at the bottom to advance your course progress.
7.1 — Four Failure Classes, Four Different Fixes · Core
| Class | Looks like | Fix |
|---|---|---|
| Prompt failure | Fails uniformly across every input type | Rewrite criteria to be explicit and testable |
| Hallucination | Confident, fluent content not grounded in any source | Grounding — retrieval, citations, verification — never a stronger instruction |
| Model mismatch | Fails only on complex, multi-step tasks; simple ones pass | Test on a higher-capability tier |
| Retrieval failure | Wrong or missing documents came back | Fix chunking, embeddings, or query construction |
The distinction the exam tests hardest: "correct documents retrieved, answer still wrong" is a hallucination/grounding failure, not a retrieval failure — retrieval already succeeded; the model is ignoring or contradicting what it was given.
7.2 — Iterating on a Fix Without Fooling Yourself · Core
- Establish a baseline eval before any change.
- Change one variable at a time — prompt or model, never both in the same test.
- Predefine the metric and the sample size before looking at results.
- Compare against the baseline with enough data to mean something.
- Regression → diagnose the root cause; don't roll back blindly (the regression might be an input-distribution shift, not the change itself).
Named anti-pattern: 50 sessions on each side, 68% vs. 62%, "deploy the winner" — two weeks later the gain evaporates. Three compounding failures: sample too small, input distribution uncontrolled, and the metric wasn't fixed before the test started. Any one of the three alone invalidates the result.
Debugging context that actually helps: the verbatim error message, the relevant code/prompt, what changed recently (a diff), and what the system is supposed to do vs. what it's actually doing. "It's broken, please fix it" gets a vague answer back — the quality of help is proportional to the quality of the context given.
⚠ Often-missed — An Out-of-Date Eval Suite Looks Like Coverage While Providing None · Gap
A prompt changes, the existing eval suite still passes — but it was built before the change and tests behavior that no longer exists in the system. This is worse than having no eval suite at all, because it produces false confidence. Keep the eval set synchronized with every prompt or system change, not just run occasionally.
7.3 — Three Optimization Levers · Core
| Lever | What it fixes |
|---|---|
| Trim context (only pass tool-output fields the next step actually needs) + cache stable prefixes | Token cost |
| Stream responses + route easy traffic to the fast tier | Latency / perceived latency |
| Batch API for non-urgent bulk work | Cost-performance at scale |
Never cut retrieval depth or context on a hunch — only after evals confirm the smaller amount holds accuracy. (If top-3 retrieved chunks score the same as top-10, cut to top-3; don't guess your way to top-1 without testing that specific change.)
Extended context sessions degrade — the model starts referencing "typical patterns" instead of the specific facts it found earlier. The fix is a scratchpad file (persist key findings outside the conversation) plus /compact to reclaim space — not restarting from scratch, which throws away everything already found.
7.4 — Common Failure Patterns to Recognize on Sight · Core
| Pattern | What happens | Fix |
|---|---|---|
| "Lost in the middle" | Model reliably reads the start/end of long input; misses the middle | Put key facts at the top; use explicit section headers |
| Access failure treated as valid empty result | A timeout is treated the same as "searched, found nothing" | Return structured error context (failure type, what was tried, partial results) — never silently return empty on success |
| Model tier creep | Work quietly routes to a pricier tier than configured | Pin model versions; set explicit spend controls |
| Aggregate accuracy masking a segment failure | 97% overall hides 60% on one document type | Validate by segment before touching review coverage |
Both extremes of subagent error handling are anti-patterns: silently returning empty as success (blocks recovery, corrupts downstream synthesis) and killing the whole workflow on one recoverable failure (throws away everything already gathered). The middle path — structured error context that lets the coordinator decide — is correct.
Exam reflexes for Domain 7
- "Correct documents retrieved, still wrong answer" → hallucination/grounding, not retrieval.
- "Fails only on the hard, multi-step cases" → model mismatch — test a higher tier.
- "Fails on every input type equally" → prompt failure — rewrite criteria.
- A/B test with 50 sessions per side, no predefined metric → invalid regardless of the result; too many uncontrolled variables.
- Eval suite passes after a prompt change but a real bug slipped through → eval suite is out of date; it looked like coverage while providing none.
- Cost rose with no usage increase → check for model-tier creep or a caching regression, not a mystery bug.
- Long agent session starts giving vague, "typical pattern" answers → context degradation; use a scratchpad file +
/compact, don't restart from zero. - Subagent timeout returned as an empty result → wrong; must return structured error context, not a silent empty success.
Test yourself on this domain. Take the Domain 7 practice quiz — 38 questions, instant scoring, an explanation for every answer.