Spring AI RAG and Vector Stores, Step by Step

TokenTextSplitter splitter = TokenTextSplitter.builder()
    .withChunkSize(500)           // tokens per chunk: tune this against your evaluation set
    .withMinChunkSizeChars(200)
    .build();
List<Document> chunks = splitter.apply(documents);

Better chunking usually comes from structure, not size. Split on headings, keep a table with its caption, keep a procedure's steps together, and put the section title into each chunk's metadata so you can show it to users or prepend it to the text.

Metadata is what makes retrieval controllable. Attach, at ingestion time, things you will want to filter or cite: the source id and title, section, document version or date, language, tenant or department, and an access-control tag. A vector store can filter on metadata with expressions such as tenant == 'acme' && year >= 2024.

Spring AI also ships transformers that call a chat model to add metadata, KeywordMetadataEnricher (keywords into excerpt_keywords) and SummaryMetadataEnricher. They help some corpora, but they cost one model call per chunk, so measure the gain before enabling them on a large corpus.

Readers cover common sources: TextReader, JsonReader, PagePdfDocumentReader and ParagraphPdfDocumentReader (artifact spring-ai-pdf-document-reader), TikaDocumentReader for Office formats (spring-ai-tika-document-reader), plus Markdown and JSoup readers in their own artifacts. All implement DocumentReader, a Supplier<List<Document>>, so a pipeline can be written as a composition: vectorStore.accept(splitter.apply(reader.get())).

Vector stores

A VectorStore stores documents with their embeddings and answers similarity queries. The interface is small: add(List<Document>), delete(...) by id list or by filter, and similaritySearch(SearchRequest). The store calls the EmbeddingModel for you, so you pass text in and get documents out.

List<Document> hits = vectorStore.similaritySearch(
    SearchRequest.builder()
        .query("How many days of annual leave do I get?")
        .topK(5)                       // return at most 5
        .similarityThreshold(0.7)      // drop weak matches (range 0 to 1)
        .filterExpression("tenant == 'acme'")
        .build());

The reference lists many backends: PgVector, Redis, Elasticsearch, OpenSearch, MongoDB Atlas, Milvus, Qdrant, Pinecone, Weaviate, Chroma, Neo4j, Cassandra, Oracle, MariaDB, Azure Vector Search, Typesense, GemFire and an S3 vector store, plus SimpleVectorStore, which the docs say is for testing and not designed for production.

For pgvector, the reference documents the starter spring-ai-starter-vector-store-pgvector. Schema creation is opt-in: you must set initialize-schema (or the constructor argument) to true. Defaults include index type HNSW and distance type cosine. The page states that PGvector supports at most 2000 dimensions for HNSW indexes, and compares index types: IVFFlat builds faster and uses less memory but has lower query performance, while HNSW has better query performance and can be built without data. Use the pgvector/pgvector image to try it locally:

docker run -it --rm --name postgres -p 5432:5432 \
  -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres pgvector/pgvector

Wiring retrieval into ChatClient with advisors

Spring AI offers two RAG advisors, from simple to modular.

QuestionAnswerAdvisor (artifact spring-ai-vector-store-advisor) is the quick start. It searches the store and appends the results to the prompt using a template:

var qa = QuestionAnswerAdvisor.builder(vectorStore)
    .searchRequest(SearchRequest.builder().similarityThreshold(0.8d).topK(6).build())
    .build();

String answer = chatClient.prompt()
    .advisors(qa)
    .user(question)
    .call()
    .content();

RetrievalAugmentationAdvisor (artifact spring-ai-rag) is a pipeline of replaceable parts, which is what you want once quality matters:

By default, ContextualQueryAugmenter does not allow an empty context: if nothing was retrieved, it instructs the model not to answer. That is a good default for a policy assistant, because "I don't have that information" is better than a guess. Call allowEmptyContext(true) only if the assistant may answer from general knowledge.

The advisor stores the retrieved documents under the key RetrievalAugmentationAdvisor.DOCUMENT_CONTEXT in both the response context and the ChatResponse metadata. That is how you build citations without searching twice.

Step by step: an HR handbook assistant

The goal is a service with two endpoints: one to ingest the handbook, one to ask a question and get an answer with sources.

Step 1: dependencies and configuration

<dependencies>
  <dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-openai</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-vector-store-pgvector</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-rag</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-jdbc</artifactId>
  </dependency>
  <dependency>
    <groupId>org.postgresql</groupId>
    <artifactId>postgresql</artifactId>
  </dependency>
</dependencies>

The OpenAI starter provides both the chat and the embedding model, and spring-ai-rag provides RetrievalAugmentationAdvisor. The Spring AI BOM (from lesson 2) manages the versions. The pgvector page lists spring-boot-starter-jdbc and the PostgreSQL driver in its manual-configuration section; they are included here so that Spring Boot configures the DataSource from spring.datasource.*.

spring:
  datasource:
    url: jdbc:postgresql://localhost:5432/postgres
    username: postgres
    password: ${DB_PASSWORD}
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      chat:
        model: gpt-4o-mini
      embedding:
        model: text-embedding-3-small      # choose from the models the OpenAI page lists
    vectorstore:
      pgvector:
        initialize-schema: true            # development only; use migrations in production
        index-type: HNSW
        distance-type: COSINE_DISTANCE
        # dimensions is taken from the EmbeddingModel when unset, per the pgvector page

Run a startup check so a mismatch between the model and the table fails fast, not at the first query: log embeddingModel.dimensions() once and compare it with the table's vector column.

Step 2: ingest

@Service
public class HandbookIngestService {

    private final VectorStore vectorStore;
    private final TokenTextSplitter splitter = TokenTextSplitter.builder()
        .withChunkSize(500)
        .build();

    public HandbookIngestService(VectorStore vectorStore) {
        this.vectorStore = vectorStore;
    }

    /** Re-ingesting a file replaces its old chunks, so edits never leave stale text behind. */
    public int ingest(String sourceId, String title, String tenant, Resource file) {
        // 1. Remove anything previously stored for this source (filter expression syntax from the reference).
        vectorStore.delete("sourceId == '" + sourceId + "'");

        // 2. Read and split.
        TextReader reader = new TextReader(file);
        List<Document> chunks = splitter.apply(reader.read());

        // 3. Attach our own metadata to every chunk (explicit, so we do not rely on splitter internals).
        List<Document> prepared = new ArrayList<>();
        for (int i = 0; i < chunks.size(); i++) {
            Map<String, Object> meta = new HashMap<>();
            meta.put("sourceId", sourceId);
            meta.put("title", title);
            meta.put("tenant", tenant);
            meta.put("chunk", i);
            prepared.add(new Document(chunks.get(i).getText(), meta));
        }

        // 4. Embed and store (the store calls the EmbeddingModel and batches the requests).
        vectorStore.add(prepared);
        return prepared.size();
    }
}

Never build the filter string from unvalidated input; sourceId here must come from your own catalogue or be strictly validated.

Step 3: query with sources

@Service
public class HandbookAssistant {

    public record Answer(String text, List<Source> sources) { }
    public record Source(String title, int chunk) { }

    private final ChatClient chatClient;

    public HandbookAssistant(ChatClient.Builder builder, VectorStore vectorStore) {
        Advisor rag = RetrievalAugmentationAdvisor.builder()
            .documentRetriever(VectorStoreDocumentRetriever.builder()
                .vectorStore(vectorStore)
                .topK(5)
                .similarityThreshold(0.5)   // a starting point; tune it with the evaluation set below
                .build())
            .build();                        // default ContextualQueryAugmenter refuses to answer with no context

        this.chatClient = builder
            .defaultSystem("You answer questions about the company handbook. "
                + "Use only the provided context. If it does not contain the answer, say so.")
            .defaultAdvisors(rag)
            .build();
    }

    @SuppressWarnings("unchecked")
    public Answer ask(String question, String tenantFromSecurityContext) {
        ChatClientResponse response = chatClient.prompt()
            .user(question)
            // Per-request filter: the tenant comes from the authenticated user, never from the question text.
            .advisors(a -> a.param(VectorStoreDocumentRetriever.FILTER_EXPRESSION,
                                   "tenant == '" + tenantFromSecurityContext + "'"))
            .call()
            .chatClientResponse();

        String text = response.chatResponse().getResult().getOutput().getText();
        List<Document> used = (List<Document>) response.context()
            .getOrDefault(RetrievalAugmentationAdvisor.DOCUMENT_CONTEXT, List.of());

        List<Source> sources = used.stream()
            .map(d -> new Source((String) d.getMetadata().get("title"),
                                 ((Number) d.getMetadata().get("chunk")).intValue()))
            .toList();
        return new Answer(text, sources);
    }
}

Returning the sources is not decoration. It lets users verify an answer, and it gives you a log of what retrieval actually returned when someone says the answer was wrong.

Step 4: run it

Start PostgreSQL (command above), export the keys, start the app, call the ingest endpoint with a text file and then ask a question that the file answers. Then ask a question that it does not answer, and confirm that the assistant says it cannot answer instead of inventing something. That second test is the one that matters.

Debugging a bad answer

When an answer is wrong, find which stage failed before changing anything.

Most of the time the fix is in the first two branches, which is why you need a way to look at retrieval directly.

Evaluating retrieval quality

Retrieval can be evaluated without a chat model, which makes it cheap and fast. Build a gold set: 30 to 100 real questions, each with the id of the source chunk or document that answers it. Then measure:

// A retrieval-only evaluation: no chat model is called. Run it with a real embedding model.
record GoldItem(String question, String expectedSourceId) { }

double hitRateAtK(VectorStore store, List<GoldItem> gold, int k) {
    int hits = 0;
    for (GoldItem item : gold) {
        List<Document> results = store.similaritySearch(
            SearchRequest.builder().query(item.question()).topK(k).build());
        boolean found = results.stream()
            .anyMatch(d -> item.expectedSourceId().equals(d.getMetadata().get("sourceId")));
        if (found) hits++;
    }
    return (double) hits / gold.size();
}

Run it while sweeping one parameter at a time: chunk size, topK, similarityThreshold. Keep the table of results. When you change the embedding model or chunker, run it again; this is your regression test.

For the answer layer, the reference documents evaluators that use a model as a judge. RelevancyEvaluator checks whether the response is relevant to the question and context, and FactCheckingEvaluator checks whether a claim is supported by a document. Both return an EvaluationResponse with isPass():

// Adapted from the reference's integration test example.
RelevancyEvaluator evaluator = new RelevancyEvaluator(ChatClient.builder(chatModel));
EvaluationRequest request = new EvaluationRequest(
    question,
    chatResponse.getMetadata().get(RetrievalAugmentationAdvisor.DOCUMENT_CONTEXT),
    chatResponse.getResult().getOutput().getText());
boolean pass = evaluator.evaluate(request).isPass();

A model judging a model is itself imperfect and costs tokens, so use it as a screening signal, sample the failures by hand, and keep the judge model fixed so scores are comparable over time.

Use cases

1. HR and policy assistant. Content changes slowly, questions repeat. Needs structure-aware chunking (policy sections), version metadata and "I don't know" behaviour.
2. Multi-tenant support knowledge base. Each customer sees only their own articles. Needs a tenant metadata filter applied on every query from the authenticated identity, never from user input.
3. On-call runbook search. Engineers ask under pressure. Needs keyword-sensitive search for error codes (hybrid search or exact-match fallback) and answers that cite the exact runbook step.
4. Contract clause lookup. Long documents with defined terms. Needs larger context per chunk, citations to page and clause, and human review before any decision.

Decision guide: is RAG the right tool?

RAG adds a store to run, an ingestion process to maintain and a search system to tune. If a few pages of text suffice, a prompt file is simpler and more reliable.

In practice: a policy assistant for an internal helpdesk

The situation. An internal helpdesk gets repeated questions about leave, expenses and equipment. The answers exist in about two hundred policy documents (assumed). The team wants an assistant that answers with a citation, and wants to know how often it is wrong before the pilot.

Starting architecture. The policies live in a document library, and the helpdesk portal is a Spring Boot application with PostgreSQL.

What changes.

1. Enable the vector extension in the existing PostgreSQL, add the pgvector starter and an embedding model, and create the store through a reviewed migration rather than initialize-schema.
2. Write an ingestion job that reads each policy, splits it by section and then by tokens, and stores sourceId, title, version and department metadata. Re-ingestion deletes by sourceId first.
3. Add the assistant from step 3 with the default no-empty-context behaviour and a system message that tells it to say when the policies do not cover the question.
4. Collect fifty real questions with the correct policy for each (the gold set), plus ten out-of-scope questions. Measure hit rate at 5, mean reciprocal rank and no-answer accuracy, and adjust chunk size and threshold until the numbers stop improving.
5. Log the question, the retrieved source ids and scores, and the user's thumbs up or down, without storing personal content longer than needed.

Result. The team should be able to report, with numbers from the gold set, how often the right policy is retrieved and how often out-of-scope questions are correctly declined. Answers show their source, so employees can verify them. Wrong answers can be traced to a stage (missing text, poor retrieval, poor generation) instead of arguing about the model. The portal gains one vector table in a database it already runs.

Architect's notes

• Retrieval is search; treat it as search. The quality levers are chunking, metadata, query rewriting, hybrid search and re-ranking. The model comes last.
• Own your evaluation set. A set of real questions with known sources is the most valuable asset in a RAG project. Without it, every change is a guess.
• Security belongs in the filter, not the prompt. If users may see only some documents, enforce that with a metadata filter derived from the authenticated identity. A system message saying "do not reveal other tenants' data" is not access control.
• Indirect prompt injection. A retrieved document can contain text that looks like instructions. Treat retrieved text as untrusted data, tell the model to use it only as reference, and keep any authority in your code.
• Freshness is an operation, not a feature. Decide how updates and deletions reach the store, how fast, and how you verify that a deleted document is gone.
• Cost has three parts. Embedding at ingestion (one-off per change), embedding the query (small, every request) and the chat call with context (the largest). More chunks in the prompt means more tokens on every call.
• Plan for model changes. A new embedding model means re-embedding the whole corpus. Keep the source documents and the ingestion code so that this is a job, not a crisis.
• Do not store what you cannot justify. Chunks and the logs of retrieved text may contain personal data. Apply the same retention and access rules as the source system.

Common mistakes

• Mixing embedding models. Ingesting with one and querying with another returns noise. Fix: configure one model, record its name with the data, and re-embed on change.
• A dimension mismatch. The table column and the model disagree. Fix: check embeddingModel.dimensions() at startup and recreate the table when the model changes.
• One chunk size for everything. A policy and a log file need different splits. Fix: choose per content type and evaluate.
• No metadata. Without source and tenant metadata you cannot cite, filter or delete. Fix: attach metadata at ingestion.
• A threshold copied from a tutorial. Score meaning depends on the model and the distance type. Fix: sweep it on your gold set.
• Allowing empty context for a factual assistant. The model then answers from general knowledge and sounds sure. Fix: keep the default refusal behaviour and test out-of-scope questions.
• Building filters from user text. String-concatenated filter expressions are an injection risk and a data-leak risk. Fix: derive filter values from the authenticated identity and validate them.
• Using SimpleVectorStore in production. The reference says it is not designed for that. Fix: use a real store.
• Evaluating only end-to-end. A wrong answer hides which stage failed. Fix: measure retrieval separately, then the answer.

Key takeaways

• RAG retrieves relevant text at query time and puts it in the prompt; most failures are retrieval failures.
• Embeddings place similar meanings near each other; use the same model for ingestion and queries, and match the store's dimensions.
• Chunk by structure first, tune size second, and attach metadata for citation, filtering and deletion.
• VectorStore offers add, delete and similaritySearch with topK, a similarity threshold and metadata filters.
• QuestionAnswerAdvisor is the quick start; RetrievalAugmentationAdvisor gives query rewriting, expansion, joining, post-processing and augmentation as replaceable parts.
• Refuse to answer without context unless general knowledge is acceptable, and enforce access control with metadata filters.
• Build a gold set and measure hit rate, rank and no-answer behaviour before and after each change.
• Debug by stage: stored, retrieved, used.

Practice exercises

Starter (about 30 minutes). Run pgvector with the Docker command, add a few paragraphs of text about a topic you know as three Documents with metadata, and call similaritySearch with three different questions. Print the text and the metadata of each result. Then ask a question the documents do not answer and look at what comes back with similarityThreshold at 0.0 and at 0.7.

Intermediate (about 2 hours). Implement HandbookIngestService and HandbookAssistant for a folder of ten text files. Return sources with every answer. Write the gold set (twenty questions plus five out-of-scope ones) and the hitRateAtK method. Try chunk sizes of 200, 500 and 1000 tokens, record hit rate at 3 and 5 for each, and pick one with a written reason.

Advanced (half a day). Add multi-tenancy and quality controls. Ingest the same files for two tenants with different content, and apply the tenant filter from a fake security context on every query; write a test proving that tenant A cannot retrieve tenant B's chunks. Add RewriteQueryTransformer or CompressionQueryTransformer for a follow-up question scenario and measure whether it improves hit rate on conversational questions. Finally, add RelevancyEvaluator over a sample of answers, and document where the judge disagrees with your own reading.

Related reading

• Spring AI: Prompting, structured output and tools — the previous lesson; tools fetch live data, RAG fetches documents.
• Spring AI: MCP and agents — the next lesson; retrieval as one tool among several.
• Spring AI: Production architecture — observability, security and evaluation in production.
• RAG systems explained — the vendor-neutral concepts.
• Building a RAG chatbot with Spring AI and pgvector — a single-post build of a similar system.
• LLM observability guide — what to track once retrieval is live.

Official documentation

• Retrieval augmented generation — QuestionAnswerAdvisor, RetrievalAugmentationAdvisor and the modular components.
• ETL pipeline — readers, TokenTextSplitter, enrichers and writers.
• Vector databases — VectorStore, SearchRequest, filters and batching.
• PGvector — starter, properties and index types.
• Embeddings and OpenAI embeddings — EmbeddingModel and provider properties.
• Evaluation testing — RelevancyEvaluator and FactCheckingEvaluator.
• Observability — vector store and embedding metrics.