CREATE EXTENSION IF NOT EXISTS vector;
The ingestion pipeline
Read your source documents, split them into retrieval-sized chunks, and store them — Spring AI embeds each chunk automatically as it writes to the VectorStore:
@Service
public class IngestionService {
private final VectorStore vectorStore;
public IngestionService(VectorStore vectorStore) {
this.vectorStore = vectorStore;
}
public void ingest(Resource file) {
var reader = new TikaDocumentReader(file);
List<Document> docs = reader.get();
var splitter = new TokenTextSplitter();
List<Document> chunks = splitter.apply(docs);
vectorStore.add(chunks);
}
}
Chunking quality caps retrieval quality — if answers are vague or miss context, revisit your chunk size and overlap before touching the model. See RAG chunking strategies for the details.
The query pipeline
The clean way to do retrieval-augmented answers in Spring AI is the QuestionAnswerAdvisor. It retrieves the top matching chunks and injects them into the prompt for you:
@Service
public class RagService {
private final ChatClient chatClient;
public RagService(ChatClient.Builder builder, VectorStore vectorStore) {
this.chatClient = builder
.defaultAdvisors(new QuestionAnswerAdvisor(vectorStore))
.build();
}
public String ask(String question) {
return chatClient.prompt()
.user(question)
.call()
.content();
}
}
Want to tune how much context is retrieved? Configure the advisor with a search request — topK (how many chunks) and a similarity threshold (filter weak matches):
new QuestionAnswerAdvisor(vectorStore, SearchRequest.builder()
.topK(5)
.similarityThreshold(0.7)
.build());
Expose it
@RestController
public class RagController {
private final RagService ragService;
public RagController(RagService ragService) {
this.ragService = ragService;
}
@PostMapping("/api/ask")
public String ask(@RequestBody AskRequest req) {
return ragService.ask(req.question());
}
}
Combine this with streaming and you have a responsive, grounded chatbot.
Production gotchas
- Dimension mismatch — the embedding model's output size must equal the configured
dimensions. Set it explicitly.
topK too high — stuffing 20 chunks wastes tokens and dilutes the answer. Start at 4–6 and tune.- No similarity threshold — without one, you always retrieve something, even for off-topic questions. A threshold lets the bot say "I don't know."
- Metadata filtering — store tenant/source metadata on each chunk so you can scope retrieval (critical for multi-tenant apps).
- Re-embedding on model change — if you switch embedding models, you must re-embed the whole corpus; vectors from different models aren't comparable.
FAQ
Is pgvector good enough for production RAG?
For most workloads (up to millions of chunks with HNSW indexing), yes — and it saves you operating a separate vector DB. Reach for a dedicated store only when you hit its scaling or feature limits.
How big should chunks be?
Start around 300–500 tokens with light overlap, then tune based on retrieval quality. See RAG chunking strategies.
Can I stream the RAG answer?
Yes — swap .call() for .stream(); retrieval happens first, then the answer streams. See streaming responses.
How do I stop it from answering outside my data?
Set a similarityThreshold and instruct the model (via system prompt) to answer only from the provided context and say when it doesn't know.
That's a complete, grounded RAG chatbot on infrastructure you already run. The framework work is small; your retrieval quality — chunking, topK, thresholds — is what makes or breaks it.
More: the Spring AI overview, RAG systems explained, and evaluating RAG with Ragas.
Building enterprise RAG and want the architecture reviewed? Let's talk.