Embeddings and pgvector Tuning for RAG

What you'll learn: what embeddings would add to the capstone's BM25 keyword search and what they would not, how to size and index a vector table in PostgreSQL with pgvector, which settings trade recall for speed, why filters behave differently on approximate indexes, and how to measure a change before you trust it. This is a design lesson. Nothing in it ran: no embedding model, no PostgreSQL and no pgvector were used in this course's environment, and every code block is pseudocode.

At a glance

Figure 1. Where a vector index would sit next to the keyword index

How to read this: the solid path is the capstone as built; the dashed red edge is the design this lesson discusses, and both indexes return chunk ids into the same store.

Embeddings help when the user's words differ from the document's words. They can hurt on exact identifiers.
The capstone has 174 chunks, so an exact scan is cheap and no approximate index is needed.
An approximate index trades recall for speed, and the trade has settings you tune against exact search.
With approximate indexes, a filter runs after the index scan, so a restrictive filter can return fewer rows than you asked for.
Add the vector path only after a measured miss says wording is the cause, and measure the change per question.

Scenario

Composite scenario, not a real company. Lumen Industrial Supply's sales engineers search product documents with the copilot. Keyword search finds "LUM-GW-200 serial ports" at once. A new hire types "gadget to watch machine tremors" and gets controller manuals, because none of those words appear in the datasheets. The head of sales asks for "AI search". The FDE must say what that would involve, what it would cost to run, and how to tell whether it worked, without building it first. Figures are illustrative.

What you will be able to do

State which kinds of retrieval failure embeddings can fix and which they cannot.
Size a vector table from the number of chunks and the embedding dimension, and check it against the index limits.
Choose between exact search, HNSW and IVFFlat, and name the settings that trade recall for speed.
Explain why filtered queries on approximate indexes can return too few rows and list the documented remedies.
Write a measurement plan that compares approximate search with exact search and the new ranking with the old, per question.

Prerequisites

Keyword, vector and hybrid search for RAG: BM25 in the capstone, where it fails, and the reciprocal rank fusion code that combines two rankings. This lesson does not repeat it.
Metadata filters and tenant-aware access control: why the capstone filters inside the query.
Evaluating retrieval and answers: per-question failure classes.
Embeddings in theory and store choices are in Data and retrieval.

Core ideas

What embeddings add to keyword search

An embedding model turns a chunk of text into a list of numbers, a vector. Texts with similar meaning get nearby vectors, so a query can match a chunk that shares no words with it. That is the one thing keyword search cannot do. It is also not free: you pick a model, pay to run it on every chunk and every query, and store and index the vectors.

Failure	Does a vector index help?	Why
Paraphrase: "machine tremors" vs "vibration"	Yes, in principle	meaning matches where words do not
Exact identifier: "LUM-GW-200"	Risk of making it worse	the Contextual Retrieval write-up says embeddings can miss crucial exact matches while BM25 handles unique identifiers
Superseded document cited	No	a metadata problem, fixed by a filter
Partner-only document shown	No	an access problem, fixed in the query
Wrong chunk size	No	chunking, not matching

That is why the design keeps BM25 and adds vectors beside it, and fuses the two rankings. Reciprocal rank fusion is covered in the search lesson and its tested code is not repeated here.

Predict first. The capstone's golden set has two misses where an expected document sits low in the plain BM25 ranking: q01 (the datasheet is at document rank 14) and q07 (an FAQ at rank 6). Would you expect an embedding model to fix both? Decide, then open the answer.

Answer

Not safely. Both misses are about words that dominate the ranking: in q01 the OPC UA terms pull in compatibility notes ahead of the sensor datasheet. Whether an embedding model would rank the datasheet higher is an empirical question that nothing in this course ran. The honest design treats it as a hypothesis, measured on those two questions and on new paraphrase questions, not as a promise.

What you store

The vector table keeps the capstone's chunk ids and the metadata the filter needs, next to the vector. The column type needs the model's output dimension, so the model choice comes first and is recorded with the data. Vectors from two different models are not comparable, so changing the model means re-embedding every chunk.

Pseudocode. Nothing below was executed.

CREATE EXTENSION vector;
CREATE TABLE chunk_vectors (
  chunk_ref   text PRIMARY KEY,        -- same ids as the capstone: "ds-vs-100#2"
  visibility  text NOT NULL,           -- public or partner_only
  status      text NOT NULL,           -- current or superseded
  skus        text[] NOT NULL,
  model_id    text NOT NULL,           -- which embedding model produced the vector
  embedding   vector(1024) NOT NULL    -- 1024 is an illustrative dimension
);

The pgvector README gives the storage cost: each vector takes 4 bytes per dimension plus 8 bytes, and each half-precision vector takes 2 bytes per dimension plus 8 (as of 2026-11-02, from the README). It also gives the limits for indexing: up to 2,000 dimensions for vector and up to 4,000 for halfvec.

Case	Calculation	Result
The capstone: 174 chunks at 1,024 dimensions	174 x (4 x 1,024 + 8) bytes	714,096 bytes, about 0.68 MiB
A customer: 30,000 chunks, same dimension	30,000 x 4,104 bytes	123,120,000 bytes, about 117.4 MiB
The same, half precision	30,000 x 2,056 bytes	61,680,000 bytes, about 58.8 MiB
A 3,072-dimension model	4 x 3,072 + 8	12,296 bytes per vector; too wide to index as `vector`

These are vector bytes only. The index, the text and the metadata are extra, so measure the real table. The point of the arithmetic is to see in minutes that the capstone is tiny and a customer's corpus is still modest, which sets how much tuning is worth doing.

Exact search first, approximate when it is too slow

By default pgvector does exact nearest-neighbour search, which the README says gives perfect recall. At 174 chunks, scanning every vector per query is trivial, so the capstone would need no vector index at all. An approximate index becomes worth its cost when the scan is too slow for your latency budget. The README says it plainly: an approximate index trades some recall for speed, and you will see different results for queries after adding one.

Figure 2. Choosing an index

flowchart TD accTitle: Choosing between exact search, HNSW and IVFFlat accDescr: Start with exact search. If it meets the latency target, stop. If not, choose HNSW when memory and build time are acceptable because it has the better speed and recall trade-off, or IVFFlat when you need faster builds and lower memory and can accept a lower trade-off. Either way, compare against exact search on real queries. A["Exact search, no index"] --> B{"Fast enough for the latency target?"} B -- yes --> C["Stop: perfect recall, nothing to tune"] B -- no --> D{"Memory and build time acceptable?"} D -- yes --> E["HNSW: better speed and recall trade-off"] D -- no --> F["IVFFlat: faster build, less memory, lower trade-off"] E --> G["Measure recall against exact search"] F --> G

How to read this: the first question is whether you need an approximate index at all; the second chooses between two with different costs.

The settings that move recall and latency

The README describes both index types and the knobs below. Defaults are as of 2026-11-02.

Index	Build-time setting	Query-time setting	What turning it up does
HNSW	`m` (default 16), `ef_construction` (default 64)	`hnsw.ef_search` (default 40)	better recall, slower build and query
IVFFlat	`lists`	`ivfflat.probes` (default 1)	better recall, slower query

HNSW builds a layered graph; the README says it has the better speed-recall trade-off than IVFFlat but slower builds and more memory, and that it can be created on an empty table. IVFFlat divides vectors into lists and searches a few of them; it builds faster and uses less memory, and it should be created after the table has some data. The README's starting points for IVFFlat are rows / 1000 lists up to 1 million rows and sqrt(rows) above that, and sqrt(lists) probes.

Pseudocode. Illustrative index statements, not run.

CREATE INDEX ON chunk_vectors USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);

BEGIN;
SET LOCAL hnsw.ef_search = 100;      -- raise recall for this query only
SELECT chunk_ref FROM chunk_vectors
ORDER BY embedding <=> :query_vector LIMIT 10;
COMMIT;

Applying the README's starting points to the customer's 30,000 chunks gives 30,000 / 1,000 = 30 lists and about 5 probes (the square root of 30 is 5.5). For a corpus of 2 million chunks the rule changes: the square root of 2,000,000 is about 1,414 lists and about 38 probes. Those are starting points to tune against exact search, not answers.

Filters on an approximate index

The capstone's access control lives inside the keyword query, so hidden chunks never enter the ranking. A vector index does not behave the same way. The README says that with approximate indexes filtering is applied after the index is scanned, so a filtered query can return fewer rows than you asked for. Its example: if a condition matches 10 percent of rows, with HNSW and the default hnsw.ef_search of 40, only 4 rows match on average.

The same arithmetic gives 0.8 rows for a filter that matches 2 percent of rows, so a role that sees a small slice of the corpus may get almost nothing back. The README lists the remedies:

iterative index scans (hnsw.iterative_scan, with strict_order or relaxed_order), which keep scanning until enough rows are found, limited by hnsw.max_scan_tuples (20,000 by default);
a regular index on the filter column when the condition matches few rows;
partial indexes when the filter has a few distinct values, and partitioning when it has many.

WARNING

Never fall back to filtering in application code after the vector search "to keep it simple". That is the post-filter design the access-control lesson measures: it returns too few results and it lets hidden chunks influence ranking. Keep the filter in the query and test for it.

Spot the flaw. A proposal says: "Create an HNSW index, run the vector search for the top 10, then drop any chunk the user may not see in Python. It is the same filter, just later." Name two problems.

Answer

First, with an approximate index the database already filters after the index scan, and removing more rows afterwards leaves fewer than 10, sometimes none. Second, hidden chunks took part in the ranking and consumed slots in the candidate list, so a user's results depend on content they may not see. The filter belongs in the query, with iterative scans or a partial index to fill the list, and a test that a public caller never sees a partner-only chunk.

Measure before you believe

Tuning is a measurement loop, not a guess. Exact search is the ground truth for the approximate index; the golden set is the ground truth for the whole ranking. Keep them separate.

Figure 3. Two comparisons, two questions

flowchart LR accTitle: Two separate measurements for a vector index accDescr: The first comparison runs each index setting against exact search on the same queries and records recall against exact and latency, to choose a setting. The second compares the new hybrid ranking against the keyword ranking on the golden set and new paraphrase questions, per question, to decide whether to ship. A["Index settings"] --> B["Approximate vs exact search"] B --> C["Recall vs exact, p95 latency"] C --> D["Choose a setting"] D --> E["Hybrid vs keyword ranking"] E --> F["Golden set plus paraphrase questions, per question"]

How to read this: the left half asks whether the index returns the same neighbours as exact search; the right half asks whether those neighbours answer the customer's questions better.

Pseudocode. The measurement loop for the index, not run.

for setting in candidate_settings:            # for example ef_search = 40, 100, 200
    for query_vector in sample_queries:
        exact  = top_k(exact_scan, query_vector, k=10)
        approx = top_k(index_with(setting), query_vector, k=10)
        recall = len(exact & approx) / 10
    record(setting, mean(recall), p95(latency), build_time, index_size)
choose the cheapest setting whose mean recall meets the target

Worked example

A design walk for Lumen's customer: 30,000 chunks, a hosted PostgreSQL they already run, 12 percent of their last 100 real queries judged "wrong results because of different wording".

Size it: 30,000 chunks at 1,024 dimensions is about 117.4 MiB of vectors, or 58.8 MiB at half precision; both are below the 2,000- and 4,000-dimension index limits.
Start with exact search. Thirty thousand vectors per query may already meet a latency target; measure it before building an index.
If it does not, build HNSW with the defaults, then raise hnsw.ef_search in steps and chart recall against exact search and p95 latency.
Keep the filter in the query and test the smallest role. If a role matches 2 percent of rows, expect 0.8 rows from the default settings and turn on iterative scans.
Fuse with the keyword ranking and compare on the golden set plus the 12 paraphrase queries, per question.

The decision rule: ship the vector path only if it repairs most of the 12 wording misses and breaks none of the questions BM25 already answers.

Assumptions: the 12 percent, the dimension and the corpus size are illustrative; nothing was measured on pgvector.
The capstone has 174 chunks (165 public and current, 5 partner-only, 4 superseded); that count is from the data, and the vector bytes in the table above are arithmetic, not a measurement.

Decision exercise. The customer already runs PostgreSQL and has 30,000 chunks and 12 wording misses in 100 queries. Options: (a) add pgvector to the database they run; (b) buy a dedicated vector search service; (c) stay on keyword search and rewrite the documents' titles. Choose and justify.

Model answer

Start with (a) as an experiment, after confirming the misses are about wording and not about missing documents. It adds no new system, exact search may be enough at this size, and filters can use the database's own metadata columns. Revisit (b) only if the measured latency or filtered-recall behaviour cannot be fixed with the documented settings. (c) is cheap and worth trying first for the worst misses, but it does not scale to every paraphrase. Whatever is chosen, measure per question, not on the average.

At the customer: "Do we need a vector database?" A good answer separates the need (paraphrase recall) from the product, says an existing PostgreSQL with pgvector is the first thing to test, gives the size arithmetic, names the filter behaviour and the operational cost, and promises a per-question measurement before any commitment.

Practice task

Write a one-page retrieval upgrade proposal for the capstone. It does not need code. Include:

the hypothesis: which of the capstone's misses you think wording causes, and which you do not;
the table layout and the filter columns, copied from the capstone's chunk fields;
the index decision (use Figure 2) with the numbers you used;
the measurement plan: ten new paraphrase questions you would add, the metric, and what result would make you abandon the idea;
the access-control tests you would add before any user sees it.

Keep the proposal honest about what you did not run.

Exercise

A customer has 80,000 chunks and a 1,536-dimension embedding model. Compute: the vector storage at full and half precision; whether vector can be indexed at this dimension under the README's limit; the IVFFlat starting values for lists and probes; and the expected number of rows for a role that matches 5 percent of rows with HNSW's default ef_search.

Acceptance criteria:

every figure comes with its formula and the README rule it uses;
you state the one assumption each result depends on;
you name which result you would check on a real table first.

Done when you have a table of the four results with formulas and one sentence on which you would measure first.

Hint 1: which formulas

Storage uses bytes per vector times the number of chunks. The README rules for lists and probes depend on whether the row count is above one million. The filter result is the default ef_search times the share of matching rows.

Hint 2: the dimension check

Compare 1,536 with the two limits in the README, one for vector and one for halfvec. Then say what that means for whether you need half precision for the index and not only for storage.

Common mistakes

Replacing keyword search with vectors and losing exact-identifier lookups.
Building an approximate index on a corpus small enough to scan, and then tuning it.
Judging a setting by latency alone, without comparing its results with exact search.
Filtering after the vector search in application code.
Forgetting that changing the embedding model means re-embedding everything.
Reporting an average improvement and hiding the questions that got worse.

Knowledge check

1. Which retrieval failures can embeddings fix, and which can they not?

They can fix paraphrase misses, where the user's words differ from the document's words, because nearby meaning gives nearby vectors. They cannot fix stale documents, access problems or bad chunking, which are metadata and design problems. They may also weaken exact-identifier lookups, so the keyword path stays.

2. Why does the capstone not need an approximate index?

It has 174 chunks, and pgvector's default exact search scans every vector with perfect recall. At that size the scan is cheap, so an approximate index would add build time, memory and a recall loss to solve a speed problem that does not exist.

3. A role sees 2 percent of the corpus and uses the default HNSW settings. What do you expect, and what do you do?

About 0.8 rows on average, from the README's arithmetic of 40 candidates times the matching share, so most queries would return nothing. Enable iterative scans so the scan continues until enough rows are found, or use a partial index or partitioning for that filter, and test the smallest role first.

4. What does "recall against exact search" measure, and what does it not?

It measures how many of the exact top-k neighbours the approximate index also returns. It does not say whether those neighbours answer the question. That is a separate measurement on the golden set, which is why the plan has two comparisons.

5. The new hybrid ranking raises the average and loses two questions. What do you do?

List the lost questions and look at why: often a weaker list pushed an unrelated document above the right one. Report the average and the losses together, and decide whether the repairs are worth the regressions for this customer. Do not ship on the average alone.

You can now

Say what embeddings add to keyword search and where they can hurt.
Size a vector table and check it against the index limits.
Choose between exact search, HNSW and IVFFlat and name the settings to tune.
Explain the filter behaviour of approximate indexes and its remedies.
Plan a per-question measurement and say what was and was not run.

Embeddings and pgvector Tuning for RAG

At a glance

Scenario

What you will be able to do

Prerequisites

Core ideas

What embeddings add to keyword search

What you store

Exact search first, approximate when it is too slow

The settings that move recall and latency

Filters on an approximate index

Measure before you believe

Worked example

Practice task

Exercise

Common mistakes

Knowledge check

You can now

Further reading

Next

Enroll to keep reading

Ask about this article

Enjoyed this? Get more like it
every Monday.

Comments

At a glance

Scenario

What you will be able to do

Prerequisites

Core ideas

What embeddings add to keyword search

What you store

Exact search first, approximate when it is too slow

The settings that move recall and latency

Filters on an approximate index

Measure before you believe

Worked example

Practice task

Exercise

Common mistakes

Knowledge check

You can now

Further reading

Next

Enroll to keep reading

Ask about this article

Enjoyed this? Get more like it every Monday.

Comments

Enjoyed this? Get more like it
every Monday.