Spring AI Production Architecture: Operate It Safely

Tracing earns its keep when a request is slow. A single trace shows whether the time went to retrieval, a tool call, one slow model call, or three model calls in an agent loop. The OpenAI and Anthropic integrations also emit HTTP-layer observations and propagate trace context, according to the reference, though streaming calls may not parent the HTTP span under the model span.

You can add your own business counters with a MeterRegistry, for example counting guardrail blocks or fallbacks, so dashboards show events that Spring AI cannot know about.

Rate limiting and cost control

Cost is a product of three things: how many requests arrive, how many tokens each carries, and which model serves them. Control all three, in layers.

Two details from the reference help. Chat memory uses a sliding window, and MessageWindowChatMemory keeps 20 messages by default, so history cost is bounded unless you raise it. And token usage is available on every ChatResponse, which lets a custom advisor enforce a budget. Spring AI's Advisor API wraps each call: advisors run in order, an advisor can inspect the request before nextCall and the response after it.

public class TokenBudgetAdvisor implements CallAdvisor {

    private final TokenBudgetService budgets;   // your service: Redis counters, database, etc.

    public TokenBudgetAdvisor(TokenBudgetService budgets) {
        this.budgets = budgets;
    }

    @Override
    public String getName() {
        return "TokenBudgetAdvisor";
    }

    // Lower order values run first (outer). The ChatClient reference says the auto-registered
    // tool-calling advisor defaults to Ordered.HIGHEST_PRECEDENCE + 300, so 100 is inner to it:
    // this advisor runs once per model call inside the tool loop, not once per request.
    // Use a value below that default to run once per request, and test whichever you choose.
    @Override
    public int getOrder() {
        return 100;
    }

    @Override
    public ChatClientResponse adviseCall(ChatClientRequest request, CallAdvisorChain chain) {
        String tenant = String.valueOf(request.context().get("tenantId"));
        if (!budgets.hasRemaining(tenant)) {
            throw new BudgetExceededException(tenant);          // mapped to HTTP 429 below
        }
        ChatClientResponse response = chain.nextCall(request);
        ChatResponse chat = response.chatResponse();
        if (chat != null && chat.getMetadata().getUsage() != null) {
            budgets.record(tenant, chat.getMetadata().getUsage().getTotalTokens());
        }
        return response;
    }
}

@ResponseStatus(HttpStatus.TOO_MANY_REQUESTS)
class BudgetExceededException extends RuntimeException {
    BudgetExceededException(String tenant) {
        super("Token budget exhausted for tenant " + tenant);
    }
}

Register it on the client and pass the tenant per request. The param call on the advisor spec is how the reference passes per-request values to advisors:

String answer = chatClient.prompt()
        .user(question)
        .advisors(a -> a.param("tenantId", tenantId))
        .call()
        .content();

Streaming calls need a StreamAdvisor implementation too. The reference's logging advisor shows the pattern, including a ChatClientMessageAggregator that combines streamed chunks so you can read usage at the end. If your service streams, implement both interfaces or the budget check will not apply to streams.

A budget that only records usage after the fact allows one overshooting request, which is usually acceptable. If it is not, estimate tokens before the call and reserve them.

For request-rate limiting, a library such as Resilience4j offers a RateLimiter with limitForPeriod, limitRefreshPeriod and timeoutDuration settings. Its annotation works on Spring beans; for per-tenant limits you will usually keep counters in a shared store such as Redis instead of one in-memory limiter per instance, because an in-memory limiter is per pod. See Rate Limiting for the algorithms.

Resilience: timeouts, retries, bulkheads and circuit breakers

Model providers fail in several different ways, and each needs a different response.

Timeouts first

Every call needs a timeout shorter than your user's patience. Provider timeouts are configurable; the Anthropic and OpenAI pages list a 60 second default for requests, which is far longer than most interactive features should wait. Set an explicit value that matches your latency budget. For MCP clients the request timeout defaults to 20 seconds (spring.ai.mcp.client.request-timeout).

Retries: know which mechanism you are using

Retry behaviour depends on the provider and the version. The OpenAI page documents two layers: a client-level spring.ai.openai.max-retries (default 3) and a separate spring.ai.retry.* group (default max-attempts of 10, exponential backoff starting at 2 seconds with multiplier 5 and a 3 minute maximum interval, and on-client-errors false so 4xx are not retried). The Anthropic page lists only spring.ai.anthropic.max-retries (default 2). Read your provider's page.

Look at those defaults with a user in mind. Ten attempts with growing waits could keep a request alive for minutes, which is far beyond an interactive timeout, and retries multiply cost and load on an already struggling provider. For a user-facing feature, lower the attempts and cap the total time, and leave long retries to background jobs.

If you need explicit control, Spring Framework 7 includes core resilience features: @Retryable and @ConcurrencyLimit, enabled with @EnableResilientMethods. @Retryable takes maxRetries, delay, multiplier, jitter, maxDelay and an includes list of exception types, so you can retry only transient errors. Never retry a tool-calling flow that performs a side effect unless the tool is idempotent.

@Configuration
@EnableResilientMethods            // turns on @Retryable and @ConcurrencyLimit
class ResilienceConfig { }

@Service
public class SummaryService {

    private final ChatClient chatClient;

    public SummaryService(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    // Bulkhead: at most 20 model calls in flight from this bean, so a slow provider
    // cannot consume every request thread.
    @ConcurrencyLimit(20)
    // ModelBusyException is your own wrapper for 429 and 5xx responses.
    @Retryable(includes = ModelBusyException.class, maxRetries = 2, delay = 200, multiplier = 2, jitter = 50)
    public String summarise(String text) {
        return chatClient.prompt().user("Summarise in three sentences: " + text).call().content();
    }
}

Circuit breakers and fallbacks

A circuit breaker stops calling a dependency that is clearly failing, which protects your threads and gives the provider room to recover. Resilience4j (with its Spring Boot 3 starter, resilience4j-spring-boot3, plus spring-boot-starter-actuator and spring-boot-starter-aop) provides @CircuitBreaker, @RateLimiter, @Bulkhead and @TimeLimiter annotations; check its documentation for compatibility with your Boot version. The fallbackMethod must be in the same class with the same parameters plus one exception parameter.

resilience4j:
  circuitbreaker:
    instances:
      llm:
        slidingWindowSize: 20           # calls considered when computing the failure rate
        failureRateThreshold: 50        # percent of failures that opens the circuit
        waitDurationInOpenState: 30s    # how long to stay open before probing again

@Service
public class AnswerService {

    private final ChatClient primary;
    private final ChatClient secondary;

    public AnswerService(@Qualifier("primaryClient") ChatClient primary,
                         @Qualifier("secondaryClient") ChatClient secondary) {
        this.primary = primary;
        this.secondary = secondary;
    }

    @CircuitBreaker(name = "llm", fallbackMethod = "viaSecondary")
    public String answer(String question) {
        return primary.prompt().user(question).call().content();
    }

    // Same parameters as answer(), plus the exception.
    public String viaSecondary(String question, Throwable cause) {
        return secondary.prompt().user(question).call().content();
    }
}

When the circuit is open, calls fail immediately and the fallback answers. The second model comes from the previous lesson; the last line of defence is a degraded answer that needs no model at all, such as search results, a cached summary, or a message that explains the feature is temporarily limited. Building Fault-Tolerant AI Systems and Resilience Patterns cover the patterns further.

Caching

Three different caches help, and they carry different risks.

1. Response cache. For questions whose answers do not depend on the user and change rarely (FAQ-style), cache the final answer. The key must include everything that affects the answer: the normalised question, the tenant, the prompt version and the model id. If you leave the tenant out, you leak one customer's answer to another.
2. Provider prompt caching. Some providers discount or accelerate repeated prompt prefixes. Spring AI's Usage interface has cache read and write token getters (available since 2.0.0; the default implementations return null), so you can check whether it is working. To benefit, put the stable material (system prompt, tool schemas, reference documents) at the start and the variable user text at the end. See Prompt Caching and your provider's documentation for exact rules.
3. Embedding and retrieval caches. Repeated queries can reuse their embeddings or retrieval results, saving embedding calls and vector store load.

@Service
public class FaqService {

    private final ChatClient chatClient;

    public FaqService(ChatClient.Builder builder) {
        this.chatClient = builder.defaultSystem("Answer using only the company FAQ.").build();
    }

    // Requires @EnableCaching and a cache manager with a time-to-live (for example Redis).
    // The key includes tenant and prompt version so a prompt change invalidates old answers.
    @Cacheable(cacheNames = "faq-answers",
               key = "#tenantId + ':' + #promptVersion + ':' + #normalizedQuestion")
    public String answer(String tenantId, String promptVersion, String normalizedQuestion) {
        return chatClient.prompt().user(normalizedQuestion).call().content();
    }
}

Do not cache personalised answers, answers built from private documents without tenant-aware keys, or anything produced by a tool with side effects. Be cautious with semantic caches that return a stored answer for a "similar" question: a wrong match returns a confidently wrong answer, so start with exact matches and measure before going further.

Secrets, guardrails and data handling

Secrets

API keys for model providers are money. Keep them out of the repository and out of YAML files: read them from environment variables (${ANTHROPIC_API_KEY}) or a secrets manager injected at deploy time. Use separate keys per environment and, where the provider allows, per service so spend is attributable. Rotate keys, and make sure no log line, trace attribute or error message includes them. Do not paste keys into prompts, test fixtures or screenshots.

Guardrails

Guardrails are checks that run before and after the model.

• Input limits. Reject oversized messages and unsupported content types before spending tokens.
• Personal data minimisation. Remove what the model does not need, such as email addresses or card numbers, before the call. A small advisor can do this. The structure follows the reference's documented BaseAdvisor example, and the regular expression is deliberately simple:

public class EmailRedactionAdvisor implements BaseAdvisor {

    private static final Pattern EMAIL = Pattern.compile("[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}");

    @Override
    public ChatClientRequest before(ChatClientRequest request, AdvisorChain chain) {
        String redacted = EMAIL.matcher(request.prompt().getUserMessage().getText()).replaceAll("[email removed]");
        return request.mutate()
                .prompt(request.prompt().augmentUserMessage(redacted))
                .build();
    }

    @Override
    public ChatClientResponse after(ChatClientResponse response, AdvisorChain chain) {
        return response;
    }

    @Override
    public int getOrder() {
        return 0;
    }
}

• Blocked topics. Spring AI provides SafeGuardAdvisor, which blocks a call when the user input contains any word from a list. The Javadoc calls it a simple case-insensitive check and a starting point: it does not handle homoglyphs, full-width characters or zero-width characters. Use it as a first filter, not as your safety system. Dedicated moderation or classification services are an option for stronger needs.
• Output validation. When you need structured data, use entity(...). The reference describes validateSchema(), which validates the JSON against the entity schema and retries automatically with error feedback, and useProviderStructuredOutput(), which sends the schema as an API-level constraint where the provider supports it.
• Tool authorisation. The model chooses tools; your code decides what is allowed. Use ToolContext for tenant and user identity (the reference says this data is not sent to the model), check permissions inside the tool, and add destructive tools per request only. The MCP and agents lesson covers this in depth.
• Prompt injection. Text from documents, web pages, emails and tool results can contain instructions aimed at the model. Treat retrieved and tool-returned content as untrusted data. Restrict tools, keep humans in the loop for consequential actions, and enforce authorisation outside the model. The retrieval side, including per-tenant metadata filters, is in the RAG lesson.

Data handling

Decide what leaves your boundary, what you store, and for how long. Chat memory stores conversations: the in-memory default is local to one JVM and lost on restart, so multi-instance deployments need a shared repository, and the reference lists JDBC, Cassandra, Neo4j, MongoDB and Redis options, some with TTL settings. Conversations hold personal data, so set retention deliberately. The reference also notes that the JDBC, Cassandra and MongoDB repositories drop tool-call messages, which affects how faithfully a conversation can be replayed. Always derive the conversation id on the server from the authenticated user, never from a client-supplied value, or one user can read another's history.

Testing and evaluation in CI

LLM features need four layers of tests, because each catches different faults.

1. Unit tests without a model. Test prompt assembly, tool methods, advisors, budget logic and redaction with ordinary JUnit. Tools are plain Java methods and advisors are plain classes.
2. Failure-path tests with a stubbed model. Prove the fallback, circuit breaker and degraded responses work, by making the client throw.
3. Contract tests against a stubbed provider endpoint. An HTTP stub confirms timeouts, 429 handling and malformed bodies behave as designed.
4. Evaluation tests against a real model. Run your golden set and fail the build below a quality bar. These are slower, cost money and vary run to run, so run them on prompt, model or retrieval changes and nightly rather than on every commit.

@Test
void usesSecondaryWhenPrimaryFails() {
    ChatClient primary = mock(ChatClient.class, RETURNS_DEEP_STUBS);
    ChatClient secondary = mock(ChatClient.class, RETURNS_DEEP_STUBS);
    when(primary.prompt().user(anyString()).call().content()).thenThrow(new RuntimeException("provider down"));
    when(secondary.prompt().user(anyString()).call().content()).thenReturn("fallback answer");

    AnswerService service = new AnswerService(primary, secondary);

    // Calls the fallback directly, because the circuit breaker annotation needs a Spring context to apply.
    assertThat(service.viaSecondary("q", new RuntimeException())).isEqualTo("fallback answer");
}

@Tag("eval")
@SpringBootTest
class RetrievalQualityEvalTest {

    @Autowired ChatClient.Builder builder;
    @Autowired VectorStore vectorStore;

    @Test
    void answersStayRelevantToRetrievedContext() {
        RelevancyEvaluator evaluator = new RelevancyEvaluator(builder);   // use a deliberately chosen judge model
        for (EvalCase c : GoldenSet.load()) {
            ChatResponse response = builder.build().prompt(c.question())
                    .advisors(RetrievalAugmentationAdvisor.builder()
                            .documentRetriever(VectorStoreDocumentRetriever.builder().vectorStore(vectorStore).build())
                            .build())
                    .call().chatResponse();
            EvaluationRequest request = new EvaluationRequest(
                    c.question(),
                    response.getMetadata().get(RetrievalAugmentationAdvisor.DOCUMENT_CONTEXT),
                    response.getResult().getOutput().getText());
            assertThat(evaluator.evaluate(request).isPass()).as(c.id()).isTrue();
        }
    }
}

The evaluation test follows the reference's RelevancyEvaluator example. Both evaluators use a model as the judge, so the reference reminds you that the results depend on the judge model and prompt: choose it deliberately and spot-check its verdicts.

Run the two groups separately in CI. With Maven's Surefire and JUnit 5 tags, that is:

mvn test -DexcludedGroups=eval     # every pull request: fast, free, deterministic
mvn test -Dgroups=eval             # nightly and on prompt or model changes: real model, a quality bar

Treat prompts as code. Keep them in files, review them, give them a version (the cache key above uses it), and require the evaluation job to pass before a prompt change merges. Choose a quality bar before looking at results, and track the pass rate over time, because slow decline is how regressions usually appear.

Deployment and operations

• Stateless services, shared state. Keep instances stateless. Conversation memory, caches, budgets and vector data live in shared stores, so any pod can serve any request and rolling updates are safe.
• Health probes must not call the model. A readiness probe that makes an LLM request costs money and turns a provider blip into a restart storm. Probe your own dependencies (database, cache); report provider health through metrics and the circuit breaker state.
• Scale on concurrency, not CPU. Model calls spend almost all their time waiting on the network, so CPU stays low while threads and connections fill. Size thread pools, connection limits and bulkheads deliberately. Virtual threads help with waiting-heavy workloads; see Spring Boot Virtual Threads.
• Graceful shutdown for streams. Enable graceful shutdown (server.shutdown: graceful) and allow in-flight responses to finish during a rollout.
• Native images need care. The tool-calling reference notes that for GraalVM native images a @Tool class must be a Spring bean or be registered for reflection. See GraalVM Native Spring Boot.
• Roll out changes gradually. Ship prompt and model changes behind a configuration flag or canary, compare token usage, latency and quality metrics before and after, and keep a one-step rollback.
• Know your limits. Providers enforce their own rate limits per key or account. Plan quota increases ahead of launches, and spread load across keys or providers only where terms allow.

In practice: a document assistant that survives a provider incident and a cost spike

The situation. An internal assistant answers employee questions from policy documents. It launches with a single model provider and a minimal Spring AI controller. In week two, the provider has a partial outage, requests start hanging for the full default timeout, and threads pile up until the whole service becomes unresponsive. In the same week one department's script sends thousands of long requests, and the monthly bill alarms finance.

Starting architecture. A controller calls ChatClient directly. There are default timeouts and default retry settings, no budget, no cache, no dashboards, and prompts live in a Java string constant.

What changes.

1. Bound the call. Provider timeout set to 10 seconds (assumed, from the interactive latency budget), retries reduced to two attempts, and a @ConcurrencyLimit bulkhead so a slow provider cannot take every thread.
2. Add a circuit breaker and fallback. A Resilience4j circuit breaker with a second, tested model as fallback. When both fail, the service returns the top retrieved passages with a short notice instead of a generated answer.
3. Add budgets. A TokenBudgetAdvisor keeps per-department token counters in Redis, and the gateway applies a per-user rate limit. The department with the script gets a clear 429 instead of consuming the shared budget.
4. Cap inputs. Maximum question length, a lower retrieval topK, and a maximum output token setting.
5. Cache stable answers. Frequently asked policy questions are cached with a key that includes the prompt version.
6. Observe. Actuator with the Prometheus endpoint, OpenTelemetry tracing at a modest sampling rate, content logging left off, and a dashboard with error rate, p95 latency, tokens by model and fallback count. Alerts fire on circuit-open and on token spend per hour.
7. Gate changes. The golden set of 40 questions (assumed) runs nightly and on any prompt change, and the build fails below the agreed pass rate.

Result. During the next provider incident, calls fail fast after the timeout, the breaker opens, the fallback model answers, and the dashboard shows the fallback count rising, so the team knows before users complain. The runaway script receives rate-limit and budget responses without affecting anyone else. Token spend becomes visible per department and per model, repeated questions stop incurring model costs, and a prompt change that lowers answer quality is caught by the nightly evaluation instead of by employees.

Architect's notes

• Order the layers by cost and certainty. Cheap, deterministic checks first; the model call last. A request rejected by a size check costs nothing.
• Fail visibly and gracefully. A degraded answer plus a metric beats a spinner and an incident. Decide for each feature what "degraded" means before launch.
• Do not build a platform too early. One service with a handful of advisors is often enough. A separate AI gateway service earns its extra hop when several applications need shared budgets, routing and key management.
• Observability has a privacy cost. Prompts and completions are user data. Default to no content logging, restrict access to any trace store that holds content, and set retention.
• Test the unhappy paths on purpose. Failure injection in a staging environment (block the provider, return 429s, return malformed JSON) proves the design. An untested fallback is a hope.
• Retries multiply load. During a real provider incident, every client retrying creates a retry storm. Keep retries few, jittered and bounded by an overall deadline.
• Costs drift. A longer system prompt, a larger topK or a new tool schema silently raises the tokens per request. Watch tokens per request over time, not just total spend.
• Own the exit. Keep prompts, golden sets and routing in your repository so that changing providers is a measured change, not a rewrite.

Common mistakes

• Leaving default timeouts and retries. Defaults can mean a minute per attempt and many attempts. Fix: set explicit timeouts and a total deadline from your latency budget.
• Logging prompts and completions in production. They contain personal and confidential data. Fix: leave the content-logging properties off, and log identifiers and sizes instead.
• Metrics labelled with user ids. High-cardinality labels overwhelm the metrics store. Fix: label by model and feature; put per-request detail on traces.
• A readiness probe that calls the model. It burns money and amplifies outages. Fix: probe infrastructure; track provider health through metrics and the breaker.
• Caching without the tenant or prompt version in the key. Answers leak across tenants or survive prompt changes. Fix: include both in the key.
• Treating SafeGuardAdvisor as complete safety. It is a simple word check. Fix: layer input limits, moderation, output validation and tool authorisation.
• Budget enforcement only in the gateway. Token cost depends on request content, which the gateway cannot see. Fix: record real usage from ChatResponse metadata in the service.
• Retrying tools with side effects. A retry can double an action. Fix: make tools idempotent or never retry them.
• No evaluation in CI. Prompt tweaks degrade quality unnoticed. Fix: golden set, written bar, nightly and on-change runs.
• Client-supplied conversation ids. One user can read another's chat. Fix: derive the id on the server.

Key takeaways

• A production AI feature is a pipeline: authenticate, limit, validate, check budget, look up cache, call the model, validate output, record and respond.
• Spring AI emits Micrometer observations for chat client calls, advisors, model calls, token usage, tools and vector stores; Spring Boot supplies the registries and tracing. Content logging is off by default and should stay off.
• Control cost with request limits, input and output caps, loop limits, model tiering, caching and per-tenant token budgets recorded from real usage.
• Bound every call with a timeout, keep retries few and for transient errors only, add a bulkhead and a circuit breaker, and keep a tested fallback plus a no-model degraded response.
• Cache only what is safe, with keys that include tenant, prompt version and model.
• Guardrails are layered: input limits, redaction, simple filters, schema validation, tool authorisation and untrusted-content handling.
• Test at four levels, and run model-backed evaluations on change and nightly with a written quality bar.
• Deploy stateless services with shared state, never probe readiness through the model, scale on concurrency, and roll out prompt and model changes gradually.

Practice exercises

Starter (about 30 minutes). Add Actuator and a Prometheus registry to a Spring AI project, expose health,metrics,prometheus, make a few ChatClient calls, and open /actuator/prometheus. Find the token usage and model-call latency series. Confirm that prompt content does not appear in any log or trace, and note which properties keep it that way.

Intermediate (about 2 hours). Implement TokenBudgetAdvisor with an in-memory TokenBudgetService, register it on a ChatClient, and pass a tenant id with .advisors(a -> a.param(...)). Write tests that show a tenant is blocked after exceeding its budget and that another tenant is unaffected. Then add a Resilience4j circuit breaker with a fallback method and prove, with a stubbed failing client, that the fallback answers.

Advanced (half a day). Build the full path for one feature: explicit timeout and a lowered retry count, a @ConcurrencyLimit bulkhead, a circuit breaker with a second provider as fallback and a no-model degraded response, a tenant-aware response cache, an email redaction advisor, Prometheus dashboards for latency, tokens and fallback count, and a tagged evaluation test with at least 30 cases. Inject failures (blocked provider, 429, slow responses) in a staging profile and write down what users and dashboards show in each case.

Related reading

• Getting Started with Spring AI — the ChatClient and configuration basics this lesson hardens.
• Prompting, Structured Output and Tool Calling — structured output and tool design.
• RAG and Vector Stores — retrieval, metadata filters and per-tenant isolation.
• MCP and Agents — tool-call limits, approvals and securing tool access.
• Choosing and Integrating LLMs — tiering, fallbacks and the evaluation harness.
• Designing the AI Request Pipeline — the layered pipeline in more depth.
• Building Fault-Tolerant AI Systems — resilience patterns for AI calls.
• LLM Observability Guide — dashboards, golden signals and incident response.
• Spring AI Enterprise Integration Guide — enterprise integration concerns.
• Spring Boot Production Mistakes — general Spring Boot operations pitfalls.
• Rate Limiting — the algorithms behind request limits.

Official documentation

• Spring AI observability
• Spring AI advisors
• ChatClient API
• Tool calling (limits, errors, observability)
• Chat memory
• Evaluation testing
• OpenAI chat (retry and timeout properties)
• Anthropic chat (timeout and retry properties)
• Spring Boot metrics
• Spring Boot tracing
• Spring Framework resilience features
• Resilience4j getting started (Spring Boot 3)
• SafeGuardAdvisor Javadoc