Enterprise AI Knowledge Platform

The problem it solves

Employees need answers from a large, mixed pile of internal documents (PDFs, wiki pages, files in object storage), and the answer has to be correct, cited and visible only to people who may see the source. Plain keyword search misses meaning; a chat model on its own makes things up. A knowledge platform has to combine both safely.

The requirements this design is built around:

• Grounded answers. Every claim cites a retrieved passage, and the system says "I don't know" when retrieval is weak.
• Tenant isolation. Business units must never see each other's documents, in search results, in cache or in logs.
• Data residency. Documents and embeddings stay inside the organisation's own cloud network.
• Predictable cost and latency. Embedding and generation are metered, so the design caches, batches and budgets them.

Architecture

The query path is synchronous and latency-sensitive. The ingestion path is asynchronous (messages on Kafka between the steps), so a slow embedding API never blocks search.

Key design decisions

pgvector inside PostgreSQL, before a separate vector database

If the organisation already runs PostgreSQL, adding the pgvector extension keeps vectors next to the rest of the data: one backup, one access-control model, one runbook, and transactional consistency between a document's metadata and its embeddings. The trade-off is scale: a dedicated vector database is built for very large corpora and heavy concurrent vector load. Start with pgvector, measure recall and latency on your own corpus, and move only when a measured limit forces it.

Hybrid search: vector and keyword together

Vector search finds meaning but can miss exact tokens such as product codes, error numbers and internal acronyms. Keyword (BM25-style full-text) search finds those exactly but misses paraphrases. Run both and merge the ranked lists with Reciprocal Rank Fusion (RRF): each document scores the sum of 1 / (k + rank) across the lists (k is commonly 60), so a document that ranks well in both rises to the top without having to compare incompatible raw scores.

// Sketch: hybrid retrieval with tenant filtering and RRF. Check the API of your Spring AI version.
public List<Document> hybridSearch(String query, String tenantId, int topK) {
    Filter.Expression tenantOnly = new FilterExpressionBuilder().eq("tenant_id", tenantId).build();

    List<Document> vectorHits = vectorStore.similaritySearch(
        SearchRequest.builder().query(query).topK(topK * 2).filterExpression(tenantOnly).build());
    List<Document> keywordHits = fullTextSearch.search(query, tenantId, topK * 2);

    return reciprocalRankFusion(List.of(vectorHits, keywordHits), topK);
}

static List<Document> reciprocalRankFusion(List<List<Document>> rankedLists, int topK) {
    final int k = 60;
    Map<String, Double> scores = new HashMap<>();
    Map<String, Document> byId = new HashMap<>();
    for (List<Document> list : rankedLists) {
        for (int rank = 0; rank < list.size(); rank++) {
            Document d = list.get(rank);
            byId.put(d.getId(), d);
            scores.merge(d.getId(), 1.0 / (k + rank + 1), Double::sum);
        }
    }
    return scores.entrySet().stream()
        .sorted(Map.Entry.<String, Double>comparingByValue().reversed())
        .limit(topK)
        .map(e -> byId.get(e.getKey()))
        .toList();
}

The tenant filter is built with the filter builder, never by concatenating a string, so a tenant id cannot change the meaning of the query.

Tenant isolation in layers

Never rely on one check.

1. The tenant is resolved from the verified token at the gateway, not from anything the client sends in the request body.
2. Every retrieval carries the tenant filter, and PostgreSQL row-level security enforces it again in the database, so a missing filter in application code returns nothing instead of everything.
3. Cache keys include the tenant (and the document version), so one tenant's cached answer can never be served to another.
4. Logs hold ids and counts, not document text.

Chunking that respects meaning

Fixed-size splitting cuts through sentences and tables. Split at structural boundaries first (headings, paragraphs), keep a small overlap, store the heading path with every chunk, and keep tables whole. Chunk size is a tunable: test two or three settings against your evaluation set instead of trusting a default.

Grounded generation

The prompt contains only retrieved passages, each with an id. The model is told to answer from them, cite the ids, and say it does not know when the passages do not contain the answer. The application then checks that every citation id exists in the retrieved set and drops or flags answers that cite nothing. Retrieval confidence has a floor: below it, the system skips generation and shows the closest documents instead.

Scaling and operations

• Embedding rate limits. Ingestion runs through a queue (Kafka) with controlled concurrency, so the embedding API's limits set the ingestion speed and never the search latency.
• Index maintenance. Build or rebuild vector indexes with CREATE INDEX CONCURRENTLY so queries keep working, and consider partitioning the vector table by tenant for large tenants.
• Cold cache. After a deploy, pre-load the cache with frequent queries instead of letting users pay for the warm-up.
• Re-embedding. Store the embedding model name and version with each vector. Changing the model means re-embedding everything, so plan it as a migration with a shadow index.

Evaluating it before trusting it

Build a small golden set: real questions, with the documents that should answer them, written by people who know the content. Measure retrieval (recall@k, mean reciprocal rank) and answer quality (is every claim supported by a cited passage?). Re-run it on every change to chunking, embedding model, fusion weights or prompt. Without this, "it feels better" is the only evidence you have.

Observability

Log, per query and without document text: tenant id, latency split (embed, search, re-rank, generate), token counts, cache hit or miss, number of passages used and a retrieval-confidence value. Dashboards show latency percentiles per stage and cost per query, and alerts fire on "no answer" rate and on citation failures.

Trade-offs to expect

Related reading

• RAG systems explained
• Spring AI course: RAG
• Spring AI course: production architecture
• Kafka scalability patterns