CCA Professional M2: Enterprise Integration & Production

Exam domains covered: D3 — Integration (19%), D4 — Evaluation, Testing & Optimization (16%)

Approximate module runtime: 158 minutes of Anthropic learning content

The Evaluation Grading Ladder

Use the cheapest reliable method for each task. Only escalate when the tier below cannot be trusted.

1. Code-based evaluation
       ↓ (when interpretation is required)
2. Model-based evaluation (LLM as judge)
       ↓ (when neither above can be trusted)
3. Human review

Code-Based Evaluation

Use when the output can be checked mechanically:

Correct JSON schema
Correct field extraction (name, date, amount)
Correct arithmetic
Exact format compliance (correct heading names, section order)
Presence/absence of required elements

No model involvement needed. Fastest and cheapest.

Model-Based Evaluation (LLM as Judge)

Use when the output requires interpretation:

Tone and brand alignment
Reasoning quality
Argument coherence
Safety and policy compliance at scale

Three required design elements for a trustworthy LLM judge:

Different model from the evaluated model — self-grading produces rationalisation, not evaluation
Constrained verdicts — "pass/fail" or "acceptable/unacceptable/escalate", never "score 1–10 with a paragraph explanation" (the judge's uncertainty gets absorbed into the score, not surfaced)
Calibrated against human-labeled outputs before use at scale — run the judge against a set of human-graded samples and confirm agreement before trusting it

The uncalibrated judge rule: An uncalibrated LLM judge is worse than no automated grade. It produces a confidence signal that looks like measurement while silently disagreeing with what humans would actually score. This is one of the most-tested traps in Domain 4.

Human Review

Reserve for:

Cases where neither code-based nor model-based can be trusted
Genuinely ambiguous outputs where interpretation requires domain expertise
Calibration exercises (grading samples the LLM judge will later use)
Regulated decisions requiring a named human decision-maker

Golden Dataset Design

The evaluation dataset must be:

Component	Why it's required
Real queries from production-like channels	Captures the true input distribution
Constructed edge cases and failure-prone scenarios	Probes the failure surface — where the system actually breaks

What makes a bad golden dataset:

Synthetic questions generated by a model from documentation → inherits the documentation's blind spots
Examples already used in the system prompt → only proves memorisation, not generalization
Questions written by the engineers who built the system → tests the builders' assumptions with the builders' assumptions

The POC-to-Production Gap

A proof of concept reliably hides four dimensions:

Dimension	What POC hides	What production reveals
Cost	Low volume, one request at a time	Cost per request × volume × months
Latency	Developer machine, no queue	p95/p99 under real concurrent load
Reliability	Single happy-path request with developer supervision	Retry storms, provider outages, cascading failures
Failure modes	Developer catches and corrects every mistake	Silent errors, hallucinated tool calls, context overflows

The exam tests this with questions describing a "successful demo." The architect's job is to name which of these four the demo didn't test, and what evidence is needed before production commitment.

Reliability Controls — Which Layer

This is one of the most-tested mechanical facts in Domain 3. Place each control at exactly its layer:

Control	Layer	What it catches
Retry	API call layer	Transient provider errors (5xx, timeout on a single request)
Circuit breaker	Service boundary	Sustained provider degradation — stops amplifying load on a struggling provider
Fallback chain	Orchestration layer	Full provider failure — gracefully degrades to a simpler path or cached response

The placement rule: Each control must be at its layer. A retry at the orchestration layer retries an entire multi-step workflow, compounding cost and side-effects. A circuit breaker at the API call layer won't help when the failure is at the service boundary.

A/B Testing Rigor

Four elements that must exist before an A/B test starts:

Pre-committed hypothesis — what you expect to change and why. Prevents p-hacking (looking at results, then deciding what to "measure")
Power calculation — minimum sample size for the effect size you care about. A test that runs for one week without a power calculation may be underpowered and produce meaningless results
One primary metric — with secondary guardrail metrics. Not "we'll see what moves"
Statistical significance gate — commit to the threshold before looking at results, not after

An offline evaluation suite gain (e.g., +6 points) doesn't guarantee a production gain. After an offline win, the next step is a controlled A/B test on a fraction of real traffic with quality and guardrail monitoring — not full deployment.

Shadow Testing vs Canary Rollout

Method	What it does	When to use
Shadow testing	New version runs on real traffic in parallel; output is logged but never shown to users	When zero user exposure is required (regulated domains, high-stakes outputs, zero tolerance for degradation)
Canary rollout	Small percentage of real traffic routes to the new version; users see its output	When you need real downstream signal and can tolerate bounded exposure risk

Shadow testing trades real downstream signal for zero exposure. It's often the only acceptable path in regulated deployments. A canary gives you real signal but requires accepting that some users see the new behavior.

Change Attribution

When a production metric moves, three causes must be distinguished before any fix is applied:

Cause	What changed	Fix direction
Model drift	Model behavior shifted (e.g. after a provider update)	Evaluate the model update; potentially pin the previous version
Data drift	Input distribution shifted (users asking different things, new document types)	Update retrieval corpus, eval set, or training data
Model-update effect	A deliberate model version change caused the shift	Attribute to the update; decide whether to roll back

Mixing these up produces the wrong fix:

Attributing data drift to model drift → rollback that doesn't help
Attributing a model-update effect to data drift → expensive corpus work that doesn't help

Leading vs Lagging Indicators

The exam tests whether you know which signals are early warning vs. post-failure:

Signal	Type	When it moves
Retrieval relevance scores	Leading	Before output quality degrades
"No grounded answer found" rate	Leading	Before users see bad answers
User complaints via support portal	Lagging	After damage is done
Monthly inference invoice	Lagging	Weeks after the behavior
Daily active users quarter-on-quarter	Lagging	Far too coarse, months after

The question pattern: "which signal gives the earliest warning of quality degradation?" → always the pipeline-internal metric.

Observability Strategy at Scale

At tens of thousands of sessions per day, full payload capture on every session is costly and still makes failures hard to find. The correct strategy:

Structured traces with correlation IDs on every session — lightweight, fast to query
Key metrics (latency, retrieval relevance, tool error rates) on every session
Full payload capture (complete prompts and responses) on a sample, plus triggered on error conditions

This way: failures are always findable (every failure triggers full capture), and routine operational cost stays low (not logging 200-token prompts and responses for every session at full volume).

Integration Patterns

MCP vs Direct API vs Agent-to-Agent

Mechanism	Use when	Key advantage
MCP server	Multiple AI applications across the org need standardised, reusable access to a system	One team maintains the server once; every current and future application reuses it
Direct API integration	A single, tightly controlled pipeline calls the model once for a deterministic task	No discovery overhead; direct and owned
Agent-to-agent protocol	Two autonomous agents from different organisations must coordinate without exposing internal systems	Each agent mediates its own org's systems across the trust boundary

MCP earns its complexity when: many systems, many consuming applications, decentralised ownership, and frequent tool churn. Each team maintains its server; every application reuses it without re-implementing the connection.

Tool Bloat and Progressive Discovery

Symptoms of tool bloat:

Tool-selection accuracy declining as new tools were added
A third of registered tools never invoked in production
Latency increasing with each added tool

Fixes, in order:

Audit and remove unused or overlapping tools
Progressive discovery — only load the tools relevant to each request, not the full catalogue
Rewrite tool descriptions to be explicit, non-overlapping, and boundary-clear

RAG Chunking Strategy

Fixed-size chunking (e.g. 300 tokens) severs clauses from the definitions and cross-references that give them meaning. The fix for document-heavy RAG over structured documents (contracts, regulations, standards):

Structure-aware chunking — aligned to clauses and sections, not character count
Metadata linking — chunk references its parent section, definitions, and cross-references
Result: retrieval returns interpretable fragments, not isolated text islands

Hybrid retrieval — for corpora with both natural-language queries and exact identifiers (part numbers, IDs, codes):

Exact identifiers → lexical/keyword matching wins
Natural-language queries → semantic embedding wins
Solution: hybrid retrieval combining both, weighted by query type detection

Cost Optimization — Order of Operations

Never apply cost levers blind. The disciplined approach:

Trace first — analyse token usage and cost per workload from production traces
Identify the dominant driver — is it context size, caching misses, output length, or model tier?
Apply targeted levers:
- Large static prefix → prompt caching
- Inflated context → context trimming (remove padding, limit retrieved chunks)
- High output length → output length controls
- All levers exhausted → evaluate model downsizing with quality measurement
Evaluate quality at each step — assume nothing, measure everything

"Switch every workload to the smallest model" applied blindly is never the right first step. First understand where the spend is.

System-Prompt Architecture — Four-Component Structure

A well-built system prompt has exactly four components, in this order:

Role definition — who is the model and what is its authority?
Testable criteria — what specific behaviors should it produce?
Boundary statements — what must it never do, and what should it say when asked?
Output format — how should results be expressed?

Non-testable (avoid)	Testable (correct)
"Be careful about security issues"	"Flag SQL injection and authentication bypass vulnerabilities"
"Always be accurate"	"For each finding, cite the verbatim clause and the regulation it violates"
"Be helpful and professional"	"Respond in JSON: {answer, confidence, sources[]}"

A criterion is testable if you can write an eval test case that passes or fails based on whether the model meets it. Underspecification is a gap the model fills with its own assumption — differently each time.

Boundary statement structure has two parts: explicit prohibition (name the specific action/topic out of scope) and refusal response (specify exactly what the model should say when triggered).

A fake or underspecified guardrail is worse than no guardrail — it creates false confidence. Templates separate "what changes" (slots) from "what must never change" (scaffolding), so filling a slot can't accidentally remove a constraint.

Few-shot examples: use 2–4 worked examples, not 10+. Always include a rejection example — a case that looks like it should trigger the behavior but should not. Examples showing only one kind of case silently bias the model toward that case.

Context Strategy Spectrum

Four strategies for managing context across a session. Production systems almost always combine them.

Strategy	Earns its place	Breaks down
Monolithic	Bounded tasks, predictable input, stable prefixes that benefit from caching	Accumulating conversations/tool loops — cost/latency scale linearly with turns
Progressive	Multi-turn dialogue, agent loops depending on recent state	Tasks needing long-range coherence or detail dropped from earlier turns
Retrieval (RAG)	Knowledge bases too large to fit; sources that change faster than redeploy	Queries requiring synthesis across many independently-scored documents
Compaction	Long-running agents where full transcript is wasteful	When summaries drop load-bearing detail (exact identifiers, numbers)

Progressive is the right default for most production workloads; monolithic is the exception for genuinely bounded tasks. Context strategy (how it reaches the model) and context sizing (how much fits) are two separate decisions.

Trimming examples:

Tool returns 6,000-token JSON; next step needs 4 fields → extract the 4 fields (~80 tokens), discard the rest
Conversation history reaches 18,000 tokens → replace with a 500-token running summary + last 3 turns verbatim
Code file from an earlier step no longer referenced → remove it from context

Four context terms not to conflate:

Term	What it is	Key fact
Context window	The model's active attention space	Resets between calls unless your application explicitly manages continuity
Retrieval	Fetched external knowledge at query time	Augments context; does not replace or expand the window
Persistent application state	Owned by your system, not the model	The model has no inherent access — requires a tool call
Summaries / memory layers	Application-managed continuity across turns/sessions	The model has zero native memory — your application stored and passed it back

Failure Modes by Architecture Type

Architecture	Breaks first	Mitigation
Agent	Unbounded tool use / growing context	Per-turn token budgets, max tool calls, stopping criteria. Eval the stopping behavior, not just output quality.
RAG	Retrieval quality drift (index not kept in sync with corpus)	Track retrieval precision/recall as its own metric, separate from output quality.
Document pipeline	No exception path for low-confidence extractions	Confidence scoring + human review queue for low-confidence cases.
Orchestrator-workers	Failure boundaries blur; a dropped subagent fails silently	Shared trace ID; reconcile coverage at synthesis (results = units submitted).

Four Logging Layers

Capture all four for every session — not just the final response:

Layer	What to log
Request	Model version, token counts, prompt ID
Response	Output tokens, latency, stop reason
Context	User role, session ID, caching hit/miss
Outcome	Did the downstream system accept the output?

Every tool invocation must be attributable. Four things, every time: who triggered it (user identity), with which permissions (access rights used), against which resource (what was accessed), and when (timestamp). Logs that only record a service-account identity provide no attribution — this is a security gap the exam asks you to identify directly.

Credential Storage — What's Allowed

Location	Correct?
Environment variables on the deployment server	Acceptable
Dedicated secrets manager (Vault, AWS Secrets Manager)	Acceptable
Hardcoded in source code / committed to git	Always wrong
Embedded in system prompt or any prompt	Always wrong
In a README or documentation file	Always wrong

Accuracy-Latency Enhancement Costs

Any enhancement that improves accuracy adds latency. A correct answer names the quality benefit, states the latency cost, and verifies the latency still fits the SLA.

Enhancement	Quality benefit	Latency cost
Add re-ranker	Improves precision — surfaces best chunks from a noisy candidate set	+100–300ms (second model pass)
Model tier upgrade	Improves reasoning quality	+100–400ms (slower model)
Increase retrieved chunks	Improves recall	Adds latency + context cost
Hybrid keyword+semantic search	Improves recall on exact identifiers	Moderate

The discipline here is to run the numbers: if p95 + the added latency still lands under the SLA, adopt and monitor. Don't reject accuracy improvements on principle.

Module 2 Trap Patterns

Trap 6 — Aggregate metric hides tail: Average accuracy of 94% looks good while a specific document type, language, or query pattern fails at 60%. Always check for tail populations before declaring a system healthy.

Uncalibrated judge (Trap variant of Trap 5): A judge that correlates 0.6 with human raters is worse than no automated grade — it produces a false sense of coverage. Calibrate before trusting at scale.

Key Rules to Memorise

Eval grading ladder: code → model (judge) → human. Use cheapest reliable method
LLM judge must use a different model, constrained verdicts, and be calibrated against human labels
Reliability controls: retry (API layer) / circuit breaker (service boundary) / fallback (orchestration)
A/B test requires: pre-committed hypothesis + power calculation + one primary metric + significance gate
Shadow testing = logged but never shown. Canary = real users see it
Change attribution: model drift vs data drift vs model-update effect — each needs a different fix
Leading indicators: retrieval relevance and grounding rates move before user-visible quality
MCP: many systems, many apps, decentralised ownership. Direct API: single pipeline, owned
System prompt: role definition → testable criteria → boundary statements → output format
Testable criterion: you can write an eval test case that passes or fails based on it
Context strategies: monolithic / progressive / retrieval / compaction — production usually combines them
Retrieval augments context; it does NOT expand the context window
The model has zero native memory — your application stored and passed it back
Agent breaks first on unbounded tool use; RAG on retrieval quality drift
Log all four layers: request / response / context / outcome
Tool attribution: who + which permissions + against which resource + when
Credential storage: env vars or secrets manager only — never source code, prompts, or README
Accuracy improvements add latency — always verify the SLA still holds after adding the enhancement

TIP

Test yourself on this domain. This module covers two exam domains — Integration and Evaluation, Testing & Optimization — 60 questions each, instant scoring, an explanation for every answer.

CCA Professional M2: Enterprise Integration & Production

The Evaluation Grading Ladder

Code-Based Evaluation

Model-Based Evaluation (LLM as Judge)

Human Review

Golden Dataset Design

The POC-to-Production Gap

Reliability Controls — Which Layer

A/B Testing Rigor

Shadow Testing vs Canary Rollout

Change Attribution

Leading vs Lagging Indicators

Observability Strategy at Scale

Integration Patterns

MCP vs Direct API vs Agent-to-Agent

Tool Bloat and Progressive Discovery

RAG Chunking Strategy

Cost Optimization — Order of Operations

System-Prompt Architecture — Four-Component Structure

Context Strategy Spectrum

Failure Modes by Architecture Type

Four Logging Layers

Credential Storage — What's Allowed

Accuracy-Latency Enhancement Costs

Module 2 Trap Patterns

Key Rules to Memorise

Enroll to keep reading

Ask about this article

Free: CCA Foundations cheat-sheet (PDF)

Enjoyed this? Get more like it
every Monday.

Comments

The Evaluation Grading Ladder

Code-Based Evaluation

Model-Based Evaluation (LLM as Judge)

Human Review

Golden Dataset Design

The POC-to-Production Gap

Reliability Controls — Which Layer

A/B Testing Rigor

Shadow Testing vs Canary Rollout

Change Attribution

Leading vs Lagging Indicators

Observability Strategy at Scale

Integration Patterns

MCP vs Direct API vs Agent-to-Agent

Tool Bloat and Progressive Discovery

RAG Chunking Strategy

Cost Optimization — Order of Operations

System-Prompt Architecture — Four-Component Structure

Context Strategy Spectrum

Failure Modes by Architecture Type

Four Logging Layers

Credential Storage — What's Allowed

Accuracy-Latency Enhancement Costs

Module 2 Trap Patterns

Key Rules to Memorise

Enroll to keep reading

Ask about this article

Free: CCA Foundations cheat-sheet (PDF)

Enjoyed this? Get more like it every Monday.

Comments

Enjoyed this? Get more like it
every Monday.