Spring AI Prompts, Structured Output and Tools

Part 2: structured output

From text to a Java type

entity(...) asks the model for JSON that matches a Java type, then deserialises it:

public enum Category { BILLING, TECHNICAL, ACCOUNT, OTHER }
public enum Priority { LOW, NORMAL, HIGH }

public record TicketTriage(Category category, Priority priority, String reason) { }

TicketTriage triage = chatClient.prompt()
    .system("Classify support emails. Use OTHER when none of the categories fit.")
    .user(u -> u.text("Email between the markers:\n<<<EMAIL\n{email}\nEMAIL>>>")
                .param("email", email))
    .options(ChatOptions.builder().temperature(0.0))
    .call()
    .entity(TicketTriage.class);

Behind the scenes, BeanOutputConverter derives a JSON Schema from the record and appends format instructions to the prompt. For generic types such as lists, use ParameterizedTypeReference:

List<TicketTriage> batch = chatClient.prompt()
    .user("Classify these emails: ...")
    .call()
    .entity(new ParameterizedTypeReference<List<TicketTriage>>() {});

Records are a natural fit: they are immutable and the field names are the contract.

Why best-effort is the right mental model

The documentation is explicit: by default the schema is appended to the prompt as text, so the model is asked to comply and the response is parsed afterwards. The converter is best-effort. A model may wrap the JSON in prose or a Markdown fence, omit a field or add one, and BeanOutputConverter is strict, so parsing then throws. You have three levels of defence, and you can combine them.

1. Provider-native structured output. spec.useProviderStructuredOutput() sends the schema to the provider as an API-level constraint, so the provider enforces it. The documentation lists support for OpenAI, Anthropic, Google GenAI, Mistral AI and Ollama (model-specific), says it is off by default because support varies, and notes that if the model's options do not support it the flag is silently ignored. Known limits include partial support for features such as $ref, anyOf and recursive types, and OpenAI does not accept a top-level array, so wrap a list in a record.
2. Schema validation with retry. spec.validateSchema() validates the reply against the schema, appends the validation error to the prompt on failure, and retries (3 attempts by default per the reference). It does not support streaming, and token usage accumulates across attempts.
3. Your own validation after you have a typed object (next section).

TicketTriage triage = chatClient.prompt()
    .user(u -> u.text("...{email}...").param("email", email))
    .call()
    .entity(TicketTriage.class, spec -> spec
        .useProviderStructuredOutput()   // request-side constraint, where the provider supports it
        .validateSchema());              // response-side check with a retry loop

Read the diagram as a bounded loop: every retry is another paid model call, which is why the number of attempts is capped.

Validate what the schema cannot

A schema proves shape, not truth. A reply can be perfectly shaped and wrong. Add checks in layers:

Bean Validation works on records:

import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.NotNull;
import jakarta.validation.constraints.Size;

public record TicketTriage(
    @NotNull Category category,
    @NotNull Priority priority,
    @NotBlank @Size(max = 300) String reason) { }

// In the service, after entity(...). "validator" is an injected jakarta.validation.Validator.
Set<ConstraintViolation<TicketTriage>> problems = validator.validate(triage);
if (!problems.isEmpty()) {
    return TicketTriage.manualReview();   // a static factory you define on the record; routes to a human
}

The schema that entity() generates is built from Jackson and Swagger-style annotations, not from Bean Validation, so these constraints are enforced only by your own validator.validate(...) call; they are not sent to the model.

Define the failure path first: what does your system do when the output is unusable? A safe default (send to a human queue, return "try again", fall back to a rule) is part of the feature, not an afterthought.

Part 3: tool calling end to end

What happens in one request

You register tools; the framework sends their definitions with the prompt; the model may answer with a request to call one; your code runs it; the result goes back; the model continues. The reference describes the loop in four steps: send the request with all tool definitions, let the model respond, execute any tool calls through ToolCallingManager and append the results to the history, and repeat until a reply arrives with no tool calls.

Every tool round trip is a full extra model call, with the whole history re-sent. Tool calls add latency and tokens; they are not free.

Step 1: write a tool

A tool is a method with a description. The description is the only thing the model sees to decide whether to call it, so write it as guidance, not a label.

import org.springframework.ai.tool.annotation.Tool;
import org.springframework.ai.tool.annotation.ToolParam;
import org.springframework.ai.chat.model.ToolContext;
import org.springframework.stereotype.Component;

@Component
public class OrderTools {

    private final OrderService orders;

    public OrderTools(OrderService orders) {
        this.orders = orders;
    }

    @Tool(description = "Look up the status of one order by its id. "
        + "Use this when the customer asks where an order is. Read-only.")
    public OrderStatus getOrderStatus(
            @ToolParam(description = "Order id such as A-1042") String orderId,
            ToolContext toolContext) {
        // The customer id comes from the server-side context, never from the model.
        String customerId = (String) toolContext.getContext().get("customerId");
        return orders.statusFor(customerId, orderId);
    }
}

// In its own file (a Java file holds one public top-level type).
public record OrderStatus(String orderId, String state, String expectedDelivery) { }

Notes on this tool:

• @Tool takes name (default: the method name), description, returnDirect and resultConverter. All parameters are required unless you mark them @ToolParam(required = false) or @Nullable.
• The ToolContext argument carries data for your code that is not sent to the model. Putting the customer id there, not in a parameter, means the model cannot ask for someone else's order.
• Return types are converted to a string, with Jackson by default. The reference lists types that are not supported for parameters or results: Optional, Future/CompletableFuture, reactive types (Mono, Flux, Flow) and functional types.
• For native image builds, the class holding @Tool methods must be a Spring bean (here @Component) or be registered for reflection.

Step 2: hand the tool to the model

String answer = chatClient.prompt()
    .system("You help customers with their own orders. Use tools for facts; never guess.")
    .user(question)
    .tools(orderTools)                                   // only for this request
    .toolContext(Map.of("customerId", authenticatedCustomerId))
    .call()
    .content();

tools(...) on the request adds to any defaultTools(...) on the client rather than replacing them. The reference notes that default tools "can also be dangerous if used carelessly" and recommends adding risk-tier or destructive tools per call, not as defaults. In Spring AI 2.0 the ToolCallingAdvisor is registered for you automatically. The reference says that supplying your own ToolCallingAdvisor suppresses the automatic one, and that exactly one tool advisor may be present in the chain.

Step 3: control the failure modes

If a tool failure message goes back to the model, make sure the message contains nothing sensitive: exception text from a database layer can include table names or values.

Step 4: test the tool as plain Java

The tool method is a normal method. Test the logic without a model at all:

@Test
void statusIsScopedToTheCustomer() {
    OrderTools tools = new OrderTools(new FakeOrderService());
    ToolContext ctx = new ToolContext(Map.of("customerId", "cust-7"));

    OrderStatus status = tools.getOrderStatus("A-1042", ctx);

    assertThat(status.state()).isEqualTo("SHIPPED");
}

Then test the surrounding behaviour (the system message, the tool wiring) with a stub ChatModel, as in lesson 2. Whether a real model chooses the right tool for a real question is a quality question; keep a small set of such questions to run on demand.

For a deeper walk-through of a single tool with the Anthropic starter, see Tool calling with Spring AI. For the difference between in-process tools and a shared protocol, see MCP vs function calling.

Three use cases

1. Support email triage (structured output, no tools). Input: an email. Output: TicketTriage. Controls: closed enums, low temperature, validateSchema(), a human queue on failure.
2. Order status assistant (one read-only tool). Input: a customer question. Tool: getOrderStatus. Controls: customer id from ToolContext, read-only, call limits, a system message that forbids guessing.
3. Invoice field extraction (structured output plus business validation). Input: OCR text of an invoice. Output: a record with supplier, total and currency. Controls: Bean Validation, a check that the total equals the sum of the lines, and a review queue when any check fails.

When not to give a model a tool

Tools are powerful and easy to overuse. Use the flowchart before you add one.

Concrete cases where a tool is the wrong choice:

• The data is known up front. If every request about an order needs the order record, load it and include it in the prompt. One model call beats two, and there is no chance of the model forgetting to ask.
• The logic is deterministic. Currency conversion, eligibility rules and date arithmetic belong in code. Let the model write the explanation, not do the calculation.
• The action is destructive or hard to undo. Refunds, deletions, account changes and emails to customers should create a draft or a request that a person or a stricter workflow approves.
• The tool is broad. A "run any SQL" tool hands the model your database. Expose narrow, named operations with typed arguments.
• Latency is tight. Each round trip adds a model call. A tool-using flow can take several times longer than a single call.

Trade-offs

In practice: an order assistant for a retail support team

The situation. A retailer's support team spends much of its day answering "where is my order?" and sorting incoming emails by topic. The team wants a triage step and a customer-facing order assistant, with no risk that the assistant exposes another customer's data or issues refunds.

Starting architecture. The order service already enforces who can see which order. Support agents read emails in a ticketing tool.

What changes.

1. Add the triage endpoint: entity(TicketTriage.class) with enums, temperature 0, validateSchema() and Bean Validation. Failures go to the existing manual queue.
2. Add OrderTools with a single read-only getOrderStatus, the customer id passed through toolContext, and the tool attached per request in the chat endpoint only.
3. Leave refunds out of the tool set. When a customer asks for one, the assistant is told in the system message to say that a person will follow up, and the endpoint creates a draft refund request for an agent to approve.
4. Cap tool calls per turn, set output limits, and turn on tool observations. Keep spring.ai.tools.observations.include-content off so arguments and results do not land in traces by default.
5. Write unit tests for the tool method and the triage service (stub model returning JSON), and a twenty-question on-demand suite for the real model.

Result. The team should see the inbox tagged with category and priority in seconds, with unusable outputs landing in the existing manual queue rather than failing silently. The chat assistant answers status questions using real data, cannot read another customer's orders because the id never comes from the model, and cannot perform actions at all. The things to watch after launch are the share of triage results that fail validation (assumed to be a small minority) and the number of tool calls per conversation.

Architect's notes

• Treat tool descriptions as part of the interface. Vague descriptions cause wrong calls or no calls. Review them like API documentation.
• Keep tools narrow and named after the business operation. getOrderStatus is safer than queryDatabase. Narrow tools are easier to scope, test and audit.
• Identity belongs on the server. Use ToolContext, the security context or a request-scoped bean for who the user is and which tenant applies. Never take them from model-supplied arguments.
• Read-only first. Ship read-only tools, watch real behaviour, and only then consider write tools, each behind approval or strict validation.
• Budget the loop. Set call limits, a timeout and token limits. A confused model can call the same tool repeatedly.
• Prompt injection is real. Text a tool returns (an email body, a web page, a document) can contain instructions. Treat tool results as untrusted data and keep authority in your code.
• Log decisions, not secrets. Log which tool ran, with which ids and how long it took. Keep arguments and results out of general logs unless you have a protected store.
• Pin and re-test models. Tool choice and JSON compliance vary by model and version, so rerun your question set when you change either.

Common mistakes

• Trusting entity() blindly. The result may be shaped correctly and still wrong. Fix: add constraints and business checks, and a failure path.
• JSON in a template. Unescaped braces break rendering. Fix: switch the delimiters with a custom StTemplateRenderer.
• Passing the user id as a tool parameter. The model can change it. Fix: use ToolContext or the security context.
• Registering write tools as defaults. Every request then carries the capability. Fix: attach risky tools per call, or do not expose them.
• Returning a Mono or Optional from a tool. These types are not supported for tool parameters and results. Fix: return a plain object and use @Nullable for optional values.
• Vague tool descriptions. The model cannot tell when to use the tool. Fix: say what it does, when to use it and whether it is read-only.
• Streaming with validateSchema(). The reference says validation is not supported with streaming. Fix: use call() for validated output.
• Putting sensitive data in thrown messages. By default a RuntimeException message is sent to the model. Fix: throw errors with safe, short messages, or customise the exception processor.

Key takeaways

• A prompt is system, user, assistant and tool messages; keep stable rules in the system message and per-request data in templates.
• entity() gives typed output but is best-effort: add useProviderStructuredOutput() where supported and validateSchema() for a bounded retry loop.
• Validate in layers: shape, constraints, business rules, authorisation and review for risky results.
• Tool calling is a loop run by your application; the model only proposes calls and never touches your systems.
• Identity and tenant come from the server (ToolContext, security context), never from model arguments.
• Keep tools narrow, read-only first, with call limits, and attach risky ones per request or not at all.
• If you can fetch the data before the call, do that instead of using a tool.
• Test tool methods as plain Java and use stubs for the model; keep a small live evaluation set.

Practice exercises

Starter (about 30 minutes). Create TicketTriage with the two enums and a reason field. Implement the triage method with a system message that lists the categories and returns entity(TicketTriage.class). Feed it five emails and note how the output varies at temperature 0.0 versus 0.8. Then write a stub-model test that returns a JSON string and asserts that your record is populated.

Intermediate (about 2 hours). Add Bean Validation to the record and a manualReview() fallback. Enable useProviderStructuredOutput() and validateSchema() on a provider that supports them and compare the number of failures over twenty crafted emails (include a few that are empty, in another language, or contain instructions such as "ignore the rules above"). Then implement OrderTools.getOrderStatus over an in-memory map, wire it with toolContext, and write the plain-Java tool test shown above.

Advanced (half a day). Add a second tool, createRefundDraft, that only creates a draft and never moves money, and attach it only when the authenticated user is a support agent. Disable the automatic loop for that endpoint with AdvisorParams.toolCallingAdvisorAutoRegister(false) and drive ToolCallingManager yourself so that a person approves each draft. Add a Micrometer timer around the loop, set a total call limit, and write down the three abuse cases you tested (prompt injection in an order note, a request for another customer's order, a loop of repeated calls) and the result of each.

Related reading

• Spring AI: Getting started — ChatClient basics and testing with a stub model.
• Spring AI: RAG and vector stores — the next lesson, retrieval instead of tools for knowledge.
• Spring AI: MCP and agents — sharing tools across processes and agent loops.
• Spring AI: Production architecture — metrics, limits and security for these features.
• Tool calling with Spring AI — a single-tool walkthrough.
• MCP vs function calling — when an in-process tool is enough.
• Prompt engineering for the enterprise — prompt design in depth.

Official documentation

• ChatClient API — templates, entity(), defaults and advisors.
• Prompts — roles, PromptTemplate and template renderers.
• Structured output converters — BeanOutputConverter and its limits.
• Provider-native structured output — request-side schemas and provider limits.
• Schema validation — validateSchema() and retries.
• Tool calling — @Tool, ToolContext, errors, limits and security notes.
• Upgrade notes — renamed classes and changed defaults.