Document Ingestion and Metadata for RAG

What you'll learn: how the capstone turns 30 product documents into searchable records, which metadata fields each retrieval decision depends on, and how to make ingestion reject a bad document instead of indexing it quietly.

At a glance

Figure 1. Ingestion: from document files to a searchable index

How to read this: dashes are the offline path every document takes; a document with bad metadata leaves at the Worker and never reaches the index. The capstone's own loader does not reject (it defaults); the checker in the Build it section does.

• Ingestion has three jobs: read the file, check its metadata, and attach that metadata to every chunk.
• Metadata that decides access or currency must be required and checked, because the retriever trusts it later.
• Reject one bad document with a reason; do not index it with a guessed default.
• Print counts after every run. A corpus you have not counted is a corpus you do not know.

Scenario

Composite scenario, not a real company. Lumen Industrial Supply, a fictional distributor, keeps 30 short product documents: datasheets, FAQs and compatibility notes. One is for partners only, one is an old revision that has been replaced, and two disagree about a battery figure. A sales assistant will answer from them. Last month a colleague added a new datasheet by copying an old one and forgot the visibility line. All counts here are illustrative except those from the capstone run shown below.

What you will be able to do

• Trace a document file through the capstone's parser to a Document and its chunks.
• Name the metadata fields a retriever needs and say which decision each one supports.
• Write a check that rejects a document with a missing or unknown field, and list every problem in one run.
• Find the two ways the capstone's tiny parser can misread a file.

Prerequisites

• Typed data models for dataclasses, which the Document and Chunk types use.
• Testing with pytest, because every claim in this lesson has a test.
• Retrieval theory (why chunk, why index) is in Data and retrieval; this page does not repeat it and covers only the hands-on ingestion step.

Core ideas

What ingestion produces

Figure 2. The document model the retriever relies on

How to read this: the three lines under DOCUMENT are the metadata each chunk inherits; a chunk carries no field of its own that the retriever filters on.

Every chunk copies visibility, status and skus from its document. That copy is what lets the search index apply a rule to one chunk without looking up its parent. The next two lessons use it: chunking splits the text, and access control reads the copied fields.

Parsing: a deliberately small parser

The capstone files start with a block of key: value lines between two lines of ---. The parser is nine lines.

Verified: capstone/fde-product-discovery-copilot/app/retrieval/chunking.py

def parse_front_matter(raw: str) -> tuple[dict[str, str], str]:
    """Tiny `key: value` parser for the block between two '---' lines."""
    if not raw.startswith("---"):
        raise ValueError("document has no front matter")
    _, block, body = raw.split("---", 2)
    meta: dict[str, str] = {}
    for line in block.strip().splitlines():
        key, _, value = line.partition(":")
        meta[key.strip()] = value.strip()
    return meta, body.strip()

str.partition splits at the first colon only, so a title such as Gateway: model 200 survives intact (as of 2026-11-02, Python documentation). The cost of being small: the parser knows nothing about quoting, lists or nesting, and every value is a plain string.

Normalising: strings become typed fields

Verified: capstone/fde-product-discovery-copilot/app/retrieval/ingest.py

def _parse_list(value: str) -> tuple[str, ...]:
    return tuple(part.strip() for part in value.strip("[]").split(",") if part.strip())


def load_documents(docs_dir: Path) -> list[Document]:
    docs: list[Document] = []
    for path in sorted(docs_dir.glob("*.md")):
        meta, body = parse_front_matter(path.read_text(encoding="utf-8"))
        docs.append(
            Document(
                doc_id=meta["doc_id"],
                title=meta["title"],
                skus=_parse_list(meta.get("sku", "")),
                visibility=meta.get("visibility", "public"),  # type: ignore[arg-type]
                status=meta.get("status", "current"),
                superseded_by=meta.get("superseded_by"),
                body=body,
            )
        )
    return docs

Two lines decide behaviour. meta["doc_id"] raises KeyError when the field is missing, which stops the run loudly. meta.get("visibility", "public") supplies a default, which does not stop anything. That asymmetry is the main lesson of this page.

Metadata design: one field, one decision

At the customer: "How do you know every document reached the index, and none reached the wrong person?" A good answer names three checks: counts reconcile against the source folder, every rejected file has a written reason, and a test asks as a public user for a partner-only fact and expects no answer.

Worked example

Ingesting ds-vs-100 (the VibraSense 100 datasheet) with the default size of 120 words and overlap of 20 gives five chunks, one per section:

That is 198 words. The body has 213 by a plain split; the 15 missing words are the five heading lines, each counted with its # mark. Headings live in the chunk's heading field and are added back to what the index sees. Over the whole folder the lesson code prints these counts from a real run:

Verified: capstone/fde-product-discovery-copilot/lessons/M3-L1/m3_l1_ingest_checks.py

def summary(docs_dir: Path) -> dict[str, object]:
    corpus = read_corpus(docs_dir)
    metas = [meta for meta, _ in corpus.values()]
    return {
        "documents": len(corpus),
        "words": sum(len(body.split()) for _, body in corpus.values()),
        "visibility": dict(Counter(m.get("visibility", "(missing)") for m in metas)),
        "status": dict(Counter(m.get("status", "current") for m in metas)),
        "documents_without_sku": sum(1 for m in metas if not parse_list(m.get("sku", ""))),
    }

Result: 30 documents, 6,251 words, 29 public and 1 partner-only, 29 current and 1 superseded, none without a SKU. Chunked with the defaults they become 174 chunks. Every number is asserted in a test, so a changed corpus fails the build instead of changing the lesson quietly.

Assumptions: the counts are the capstone's fictional data; the word count is str.split on the body after the front matter, a plain approximation and not a token count.

Build it

Work in capstone/fde-product-discovery-copilot. The lesson folder lessons/M3-L1/ holds a checker that keeps the capstone's parser but refuses to guess.

Verified: capstone/fde-product-discovery-copilot/lessons/M3-L1/m3_l1_ingest_checks.py

def problems_in(meta: dict[str, str], known_skus: set[str] | None = None) -> list[str]:
    """Problems with one document's metadata. An empty list means it may be indexed."""
    found = [f"missing '{key}'" for key in REQUIRED if not meta.get(key)]
    if meta.get("visibility") and meta["visibility"] not in VISIBILITIES:
        found.append(f"unknown visibility '{meta['visibility']}'")
    status = meta.get("status", "current")
    if status not in STATUSES:
        found.append(f"unknown status '{status}'")
    if status == "superseded" and not meta.get("superseded_by"):
        found.append("superseded but superseded_by is empty")
    if status == "current" and meta.get("superseded_by"):
        found.append("current but superseded_by is set")
    if known_skus is not None:
        found += [f"unknown SKU {s}" for s in parse_list(meta.get("sku", "")) if s not in known_skus]
    return found

Run it and the tests:

./.venv/Scripts/python lessons/M3-L1/m3_l1_ingest_checks.py
./.venv/Scripts/python -m pytest lessons/M3-L1 -q

On the real corpus the check reports nothing wrong, and the tests prove it also catches a missing visibility, an unknown status, a dangling superseded_by and an unknown SKU, all in one pass. One test shows the capstone loader turning a document without visibility into a public one; that is the behaviour to avoid in production.

Exercise

Spot the flaw. Read the load_documents excerpt again and find two ways a document can be misread without any error.

Task: extend problems_in so a SKU that does not match the pattern LUM-XX-999 (two capital letters, three digits) is reported as malformed SKU .... Acceptance criteria: the real corpus still reports nothing; a document with sku: [lum-vs-100] is rejected; the message names the SKU. Done when a new test in lessons/M3-L1/ passes and the whole capstone suite stays green.

Common mistakes

• Defaulting a missing access field instead of rejecting the document.
• Indexing first and discovering a bad document from a wrong answer later.
• Parsing metadata with a hand-made parser and never testing a value that contains a colon or a dash run.
• Dropping status and superseded_by, so old revisions compete with current ones.
• Not printing counts, so a silently skipped file is never noticed.

Knowledge check

You can now

• Describe how a file becomes a Document and then chunks that carry its metadata.
• Choose which metadata fields to require and defend each choice by the decision it supports.
• Write and test a checker that rejects bad documents with reasons.
• Explain two ways a small parser can misread a file.

Further reading

• Python documentation, str.partition, the method the front-matter parser uses.
• PostgreSQL documentation, Full text search: introduction, which describes reducing each document to normalised lexemes before searching.
• Optional next step: Keep RAG data current, sync, real deletes and what to do with scans and tables.

Last verified: 2026-11-02

Next

Chunking trade-offs