CCA Associate Foundations course · Page 8 of 10 · ← Back to all courses · Weight: 10%. Mark this page complete at the bottom to advance your course progress.
7.1 — Four Failure Classes, Four Different Fixes · Core
| Class | Looks like | Fix |
|---|---|---|
| Prompt failure | Fails uniformly across every input type | Rewrite criteria to be explicit and testable |
| Hallucination | Confident, fluent content not grounded in any source | Grounding — retrieval, citations, verification — never a stronger instruction |
| Model mismatch | Fails only on complex, multi-step tasks; simple ones pass | Test on a higher-capability tier |
| Retrieval failure | Wrong or missing documents came back | Fix chunking, embeddings, or query construction |
The distinction the exam tests hardest: "correct documents retrieved, answer still wrong" is a hallucination/grounding failure, not a retrieval failure — retrieval already succeeded; the model is ignoring or contradicting what it was given.
7.2 — Iterating on a Fix Without Fooling Yourself · Core
- Establish a baseline eval before any change.
- Change one variable at a time — prompt or model, never both in the same test.
- Predefine the metric and the sample size before looking at results.
- Compare against the baseline with enough data to mean something.
- Regression → diagnose the root cause; don't roll back blindly (the regression might be an input-distribution shift, not the change itself).
Named anti-pattern: 50 sessions on each side, 68% vs. 62%, "deploy the winner" — two weeks later the gain evaporates. Three compounding failures: sample too small, input distribution uncontrolled, and the metric wasn't fixed before the test started. Any one of the three alone invalidates the result.