What you'll learn: how to ask a model for JSON and still not trust it. You derive a JSON schema from a Pydantic model, validate every reply, classify what can go wrong, make exactly one bounded repair attempt, return a clear error when that fails, and measure what the repair costs. The capstone does its validation with Pydantic itself, not with the
jsonschemapackage.
At a glance
Figure 1. Validate, repair once, then fail
How to read this: two checks and one repair; the failed state is a normal outcome the API reports.
- A model reply is text: treat it as untrusted input, like a request body.
- A Pydantic model gives you the type, the validation and the JSON schema at once.
- A reply fails in two ways: not JSON, or JSON of the wrong shape.
- Retry once with the problem stated; never loop. Each retry is a billed call.
- Even a provider's constrained-output feature does not remove the need to validate.
Scenario
Composite scenario, not a real company. Lumen Industrial Supply turns each incoming request into a requirements record: category, quantity, budget, protocols. The next steps (search, compatibility checks) need exact fields. During a pilot, a few model replies come back with a sentence before the JSON, one with a negative quantity, and one with the budget as a string. The team asks what the service should do with each. All figures are illustrative.
What you will be able to do
- Derive a JSON schema from a Pydantic model and read its constraints.
- Classify a reply as valid, not JSON or wrong shape, and say what Pydantic does by default with strings and extra keys.
- Implement and test one bounded repair retry, and compute what it adds to the cost.
Prerequisites
Read Typed data models: dataclasses and Pydantic for field constraints and Prompts and untrusted input for why a reply is untrusted. The architecture course has no page on this topic, so everything is here.
Core ideas
One model, three jobs
Write the shape once as a Pydantic class; it validates replies and produces the JSON schema you send with the request. The capstone's Requirements has every field optional, with constraints on two.
Verified: capstone/fde-product-discovery-copilot/app/models.py
class Requirements(BaseModel):
"""What the customer asked for, as structured data. Every field is optional."""
category: Category | None = None
quantity: int | None = Field(default=None, gt=0)
budget_eur: float | None = Field(default=None, gt=0)
protocols: list[str] = Field(default_factory=list)
environment: list[str] = Field(default_factory=list)
must_have: list[str] = Field(default_factory=list)
nice_to_have: list[str] = Field(default_factory=list)
keywords: list[str] = Field(default_factory=list) # topic words, for example "vibration"
skus: list[str] = Field(default_factory=list) # products named in the request (resolved in code)Requirements.model_json_schema() returns a plain dict. As Pydantic documents, gt=0 becomes exclusiveMinimum: 0, an optional field allows null, and a field with a default is not required. The quantity property from a real run:
{"anyOf": [{"exclusiveMinimum": 0, "type": "integer"}, {"type": "null"}], "default": null, "title": "Quantity"}| Concern | Pydantic here | Rough Java equivalent |
|---|---|---|
| Declare the shape | a class with typed fields | a record or class |
| Constraints | Field(gt=0, pattern=...) |
Bean Validation annotations |
| Parse and validate JSON | model_validate_json, or json.loads then model_validate |
Jackson, then a validator |
Three things can happen to a reply
Predict first. Does Pydantic accept each reply for Requirements: (a) {"quantity": "40"}, (b) {"quantity": 40.5}, (c) {"quantity": 3, "ignore_the_rules": true}?
Answer
(a) Accepted, and quantity becomes the integer 40: Pydantic's default lax mode converts a numeric string. (b) Rejected: a float is accepted for an int only without a fractional part. (c) Accepted, and the extra key is silently dropped by default. Strict mode and extra="forbid" change (a) and (c).
| Reply text | Verdict | Why |
|---|---|---|
{"category": "sensors", "quantity": 3} |
ok | parses and matches |
```json ... ``` fenced |
ok | the capstone strips a Markdown fence first |
Sure! Here you go: {...} |
not_json | prose around the JSON cannot be parsed |
{"quantity": -4} |
wrong_shape | gt=0 fails |
{"category": "toasters"} |
wrong_shape | not one of the allowed categories |
[1, 2, 3] |
wrong_shape | valid JSON, but not an object |
The capstone treats both failures alike, because json.JSONDecodeError is a subclass of ValueError and Pydantic raises ValidationError for the rest. Still log the distinction: a rise in not_json points at the prompt or truncation; a rise in wrong_shape points at the schema.
As of 2026-11-02 the Anthropic documentation describes a mode that constrains generation to a schema, and also says the output may still not match it in some cases (a refusal, or a reply cut off at the token limit) and that some constraints, such as numeric bounds, are not supported. The capstone's reference adapter does not use it: it appends the schema to the system message as text and has never been run against the live API. Validate in your code either way.
The bounded repair
Figure 2. The request with its single repair