CCA Domain 4: Prompt Engineering and Structured Output

CCA Foundations course · Page 5 of 8 · ← Back to all courses · Weight: 20%. Mark complete at the bottom to advance.

4.1 — Explicit Criteria Over Vague Instructions · Core

• Use categorical inclusion + exclusion lists ("flag logic bugs, security vulnerabilities, resource leaks; do NOT flag formatting, naming preferences, or patterns used consistently elsewhere") — not "be conservative" or "only report important issues." Vague hedges don't define what counts, so they don't reduce false positives; the model still has to guess where the line is, and it guesses differently every run.
• False-positive trust cascade: one noisy category erodes trust in every category, including the accurate ones — once a developer has been burned by a false "security vulnerability" flag, they start skimming or ignoring every finding, even correct ones. → Disable the bad category first to restore trust, then fix its prompt separately rather than tuning it live in front of users who are already tuning out.
• Severity calibration needs concrete code examples per level, not just adjective labels — a "medium" severity example next to a "high" one is what produces consistent classification.

⚠ Often-missed — Confidence Filters Don't Reduce False Positives · Gap

"Only report high-confidence findings" sounds like a precision fix, but confidence and correctness aren't the same axis — the model can be highly confident about a finding that's still a false positive because the underlying category itself is too broad. The exam distractor pairs a confidence-threshold answer against the correct one (explicit report/skip categories); confidence filtering is the wrong tool for a categorical boundary problem.

4.2 — Few-Shot Prompting · Core

• Use 2–4 targeted examples, reserved for ambiguous cases — added when detailed prose instructions alone still produce inconsistent output.
• Show the reasoning, not just the answer — why one action was chosen over a plausible alternative, especially for ambiguous tool selection or coverage-gap calls. An example that only shows the final label teaches pattern-matching on surface features; one that shows the reasoning teaches the underlying judgment, which is what transfers to cases the examples didn't cover.
• Five useful types to mix: ambiguous (a genuinely hard judgment call), format (the exact output shape expected), positive-negative (a case that should trigger vs. a lookalike that shouldn't), varied-structure (different document layouts — inline citations vs. bibliographies, narrative vs. tables — matters most when production documents genuinely vary in shape, to cut hallucinated/empty extraction), informal (messy real-world phrasing, not textbook input — matters most when live input skews far from how a spec would phrase it).
• Examples generalise to novel patterns; a longer rulebook of prose does not — that's why few-shot beats "add more instructions" once instructions alone have plateaued.
• Order matters less than coverage. Four examples spanning the five types above beat four examples of the same type repeated — each additional example should teach the model something the others didn't, not reinforce a pattern it already handles well. Padding a prompt with near-duplicate examples burns context budget without buying any additional generalisation.

4.3 — JSON Schemas & Structured Output · Core

• Tool use with a JSON schema is the most reliable way to guarantee schema-compliant output — via Pydantic + client.messages.parse(), or a raw schema through output_config.format.
• Strict schemas eliminate syntax errors but NOT semantic errors — line items can still fail to sum to the total, or a value can land in the wrong field. Semantic errors need validation (4.4), not a stricter schema. This is a frequently tested distinction: "the output is always valid JSON but sometimes wrong" is a semantic-error scenario, not evidence the schema enforcement failed.
• Nullable fields + remove them from required → prevents the model fabricating a value to satisfy a schema field the source simply doesn't contain.
• Add an "unclear" enum value for genuinely ambiguous cases, and an "other" + free-text detail pattern for extensible categories the schema can't fully enumerate.
• Self-correction fields: extract stated_total and calculated_total, plus a conflict_detected boolean, so internal inconsistencies surface instead of silently passing schema validation.
• tool_choice: "any" guarantees some structured tool call when several extraction schemas exist and the document type is unknown up front; {"type": "tool", "name": "..."} forces one specific tool (e.g. metadata extraction before enrichment).
• Put format-normalisation rules in the prompt itself, alongside the strict schema — the schema constrains shape, but inconsistent source formatting (dates as "03/14/2024" vs "March 14, 2024," amounts with or without currency symbols) still needs prose instructions telling the model how to normalise before it fills the schema.

4.4 — Validation, Retry & Feedback Loops · Core

• Retry-with-error-feedback: on a validation failure, resend the original document + the failed extraction + the specific validation error so the model can self-correct — not a bare "try again," which gives the model no new information and often reproduces the identical mistake.
• Know when retries can't help. Retries fix format/structural errors. They cannot conjure information that's simply absent from the source (e.g. a figure that only exists in a different document you never provided) — check whether the gap is structural or a genuine absence before looping.
• detected_pattern on each finding lets you analyse afterward which code constructs trigger false positives when developers dismiss them, feeding back into 4.1's criteria — this is how a static rulebook stays current instead of drifting further from reality release after release.
• A validation failure is not automatically a model failure. Before retrying, check whether the validator itself is wrong (an overly strict regex on a legitimately varied date format) — retrying against a broken validator just burns tokens reproducing the same "failure."

⚠ Often-missed — Multi-Instance Verification Architecture (VIP routing) · Gap

• Pass 1 generates; Pass 2 is an independent instance that verifies and outputs a confidence score — it must have zero prior context from the generation pass, or the check inherits the same bias it's meant to catch.
• Route on that score: send / human_review / regenerate.
• Use tiered thresholds per customer segment rather than one global cutoff — a VIP or high-value segment routes to human review at a higher confidence threshold than a low-stakes segment, because the cost of an error differs by segment even when the model's confidence doesn't.
• Regenerate is not the same fallback as retry (4.4). Regeneration re-runs the whole generation pass from the source when the verification pass flags a problem; retry re-runs extraction when a schema/format validation fails. Confusing the two triggers the wrong recovery path.

4.5 — Message Batches API · Core

• 50% cost savings, up to a 24-hour processing window, NO latency SLA (most finish within an hour, but that's not guaranteed), NO multi-turn tools (can't execute a tool mid-request and feed the result back in). If a task genuinely needs a tool loop, it belongs on the synchronous API regardless of volume or cost pressure — batching can't be retrofitted onto a workflow that needs to react to a tool's own output mid-request.
• custom_id correlates each request with its response and enables resubmitting only the failed documents — keep a custom_id → document mapping.
• Use the synchronous API whenever someone is waiting on the result (pre-merge checks) or tool loops are needed; reserve batching for non-blocking, latency-tolerant jobs (overnight reports, weekly audits). Don't move a workflow a developer is blocked on onto batching just because a bigger batch job is being scheduled at the same time.
• Refine the prompt on a small representative sample first (e.g. ~50 docs — a fraction of a percent of full-batch cost) before processing large volumes — a prompt bug caught on 50 documents is cheap; the same bug caught after a 10,000-document batch means resubmitting the whole run.
• SLA sizing: max_gap = SLA − max_batch_duration. A 36-hour SLA against a 24-hour worst-case batch window leaves 12h of slack — submitting every 8–10h keeps every document inside the SLA with margin. Skipping this arithmetic and submitting only once a day risks a document that arrives just after the daily cutoff missing the SLA entirely.

4.6 — Prompt Chaining & Multi-Instance Review · Core

• Attention dilution beyond ~14 files → split into per-file passes + a cross-file integration pass — a bigger context window does not fix quality that's degraded by breadth, not size.
• Self-review is structurally biased, not a reasoning-depth problem — the generation reasoning sits in the session's own context and anchors it toward approving its own choices. Extended thinking does NOT fix this, because it deepens reasoning within that same biased context. Only a fresh instance with no prior context removes the bias.
• Pydantic retry only for format/structural errors — NOT when data is simply absent from the source (see 4.4).
• A verification pass where the model self-reports a per-finding confidence score enables calibrated routing of uncertain findings (see Domain 5.5).
• Multi-pass and multi-instance solve different problems — multi-pass (per-file + integration) fixes breadth (too much material for one pass to hold consistently); multi-instance review fixes bias (the same instance grading its own work). A large single-file change with subtle bugs needs a fresh reviewer, not more passes; a 40-file change needs both.

Exam reflexes for Domain 4

• "Absent data causing retry loop" → nullable fields + remove from required (schema); retries can't invent missing source data (4.4).
• "Conflicting / can't-sum totals" → self-correction fields (calculated_total, conflict_detected).
• "High volume, can wait" → Batches API (50% / 24h / no SLA / no tools).
• "Blocking pre-merge check" → stay on the synchronous API, even if a batch job runs alongside it.
• "Self-review bias" → independent instance, zero prior context — not extended thinking.
• "Misrouting" → fix descriptions, not few-shot (Domain 2).
• "Over-flagging / vague criteria" → explicit report vs. skip categories, not "be conservative."
• "Detailed instructions still inconsistent" → add 2–4 few-shot examples showing reasoning + format.
• "SLA for continuous document arrival" → max_gap = SLA − max_batch_duration, submit within that gap.
• "14+ file review" → per-file passes + integration pass.
• "Confidence-only filtering to cut false positives" → wrong axis — use categorical report/skip criteria instead (4.1).
• "Inconsistent output despite detailed prose" → 2–4 few-shot examples covering varied cases, not more prose.
• "Extraction ignores format-normalisation prompt but schema is strict" → keep normalisation rules in the prompt alongside the schema — the schema alone can't teach formatting.
• "Large single-file change with subtle bugs" → independent review instance (bias), not more passes (breadth).
• "Verification flags an issue after generation" → regenerate from source, distinct from a retry against a schema/format failure.