Keep RAG Data Current: Sync, Deletes, Messy Files

The plan summary is added 1, changed 2, removed 1, unchanged 27, and the index ends with 174 - 5 + 1 + 1 = 171 chunks. A public search for "software licensing" no longer returns the FAQ.

At the customer: "How do we know a deleted document is really gone?" The plan lists the removal, the chunk count drops by that document's chunks, and a test searches for a phrase only it contained and expects nothing.

Messy sources

load_documents reads .md files only, so scans, tables and images need an extraction step before the sync.

Figure 3. The extraction step before the sync

How to read this: every piece carries a source and a region; an unknown file type is an error, not a skip.

Verified: capstone/fde-product-discovery-copilot/lessons/M3-L9/m3_l9_extract.py

@dataclass(frozen=True)
class Piece:
    text: str
    source: str  # file name, kept so the citation can name it
    region: str  # "block 2", "row 5", "line 3", "figure": where in the file
    kind: str  # one of KINDS
    confidence: float | None = None  # set for OCR only

    @property
    def citation(self) -> str:
        return f"{self.source}, {self.region}"

The Extractor protocol is one method, extract(path) -> list[Piece]. The lesson ships four: plain text (a piece per block), CSV (a piece per row, header repeated), image captions (a person writes figure.png.caption.txt; the caption is indexed, not the image) and OcrExtractor, built only when an OCR package imports. There is no PDF extractor: pypdf is not in requirements.txt, and this lesson does not add it.

Verified: capstone/fde-product-discovery-copilot/lessons/M3-L9/m3_l9_extract.py

def triage(pieces: list[Piece], min_confidence: float = 0.9) -> Triage:
    """OCR pieces below the threshold, or without a score, and empty pieces go to a person, not to the index."""
    accepted, review = [], []
    for piece in pieces:
        low = piece.kind == "ocr" and (piece.confidence is None or piece.confidence < min_confidence)
        (review if low or not piece.text.strip() else accepted).append(piece)
    return Triage(tuple(accepted), tuple(review))

The 0.9 is illustrative: set yours by reading a sample. Accepted pieces become one document with the region as a heading, so a hit can cite the page or figure:

Verified: capstone/fde-product-discovery-copilot/lessons/M3-L9/m3_l9_extract.py

    body = "\n\n".join(f"## {p.region}\n\n{p.text}" for p in pieces)
    return Document(doc_id, title, skus, visibility, "current", None, body)

The middle column is this course's suggestion, not a sourced fact.

What the primary documentation says (as of 2026-11-04; check before relying):

• pypdf: an image-only page "may be minimal or visually empty", "pypdf is not OCR software", and "tables are just absolutely positioned text" (extract text).
• Tesseract, another OCR engine: "does not support reading PDF files" (input formats); it works best at 300 DPI or more and has a known problem with tables without custom layout analysis (quality guide).
• RapidOCR, the optional engine here: a call returns boxes, txts and scores, one per line, and you install an inference engine such as onnxruntime yourself (usage, install).

Embedding the image itself: not built here

Multimodal retrieval embeds the image as a vector, so a text query finds a picture by similarity instead of finding text read out of it. CLIP is one published example: trained on 400 million (image, text) pairs to predict which caption goes with which image (Radford et al., 2021). The capstone does none of this: it has no embedding model, only keyword search, and none was run for this page.

Build it

Work in capstone/fde-product-discovery-copilot. The lesson folder lessons/M3-L9/ holds m3_l9_sync.py, m3_l9_extract.py and their tests. Run:

./.venv/Scripts/python -m pytest lessons/M3-L9 -q

Read test_removed_document_can_no_longer_be_retrieved and test_add_only_sync_still_serves_the_deleted_document side by side: the second one empties the plan's removed list before applying it, which is the whole difference.

The extraction tests need no OCR package: they use a fake engine and a PNG made with the standard library; one test reads a real image and skips itself without the packages. To read a real image, install the optional packages into the .venv only (requirements.txt is unchanged) and run the demo, which draws two images with Pillow:

./.venv/Scripts/python -m pip install -r requirements-ocr.txt
./.venv/Scripts/python lessons/M3-L9/m3_l9_ocr_demo.py

Output from a run on 2026-11-04 (Windows, Python 3.14.2, rapidocr 3.10.0, onnxruntime 1.31.1; RapidOCR's own log lines are left out):

clean.png accepted 3 (3 ocr), set aside for review 0
  clean.png, line 1: 'Part LUM-VS-100 torque 12 Nm' score 0.992
  clean.png, line 2: 'Seal limit 90 degrees C' score 0.995
  clean.png, line 3: 'Order code LUM-0B5-17' score 0.996
degraded.png accepted 3 (3 ocr), set aside for review 0
  degraded.png, line 1: 'Part LUM-VS-100 1orque 12 Nm' score 0.958
  degraded.png, line 2: 'Seal limit 90 degrees C' score 0.983
  degraded.png, line 3: 'Order code LUM-0B5-17' score 0.976

In the degraded image "torque" was read as "1orque" with a score of 0.958, above the 0.9 threshold. A high score is not proof of a correct reading, so a person still checks a sample of accepted pieces, especially part numbers. Three further runs here printed identical scores; other versions or platforms may not.

Decide first, then reveal.

Exercise

Task: add describe_change(old: Document, new: Document) -> list[str] that names which fields differ, for a log line such as visibility public -> partner_only. Acceptance criteria: a body-only edit returns ["body"]; a label change returns the message above; identical documents return an empty list. Done when new tests in lessons/M3-L9/ pass and the full capstone suite stays green.

Common mistakes

• Syncing add and replace only, so deleted documents stay searchable.
• Hashing the body only, so a change of access label is never re-indexed.
• Replacing chunks by id, which leaves orphans when a document gets shorter.
• Letting an empty or unmounted source delete the whole index.
• Indexing OCR text without its source, region and score, or treating a high score as proof of a correct reading.

Knowledge check

You can now

• Build a fingerprint that covers text, access label and version.
• Plan and apply an incremental sync, with deletes that remove retrievable chunks.
• Add a guard that refuses a suspicious delete.
• Extract scans, tables and captions into cited pieces, and set weak OCR aside.
• Say what multimodal retrieval is, and that this capstone does not do it.

Further reading

• Python, hashlib: the fingerprint's SHA-256.
• pypdf, Extract text: limits with scans and tables.
• Tesseract, Improving quality: DPI and known limits.
• RapidOCR, Usage: the result fields.

Last verified: 2026-11-03

Next

Back to the core path: Context and citations