Skip to content
    All guides

    Forward Deployed Engineer

    Prompt, RAG or Fine-Tune: A Decision Guide

    Choose between a better prompt, RAG, structured output and fine-tuning with a decision table, a break-even sketch and three Lumen cases the reader decides.

    By Published 9 min read

    • fde
    • fine-tuning
    • rag
    • prompting
    • evaluation
    • cost
    Share

    This lesson is part of Forward Deployed Engineer: RAG, Search and Data. Enroll for free, then sign in, to read it in full.

    Enroll to unlockAlready enrolled?

    What you'll learn: how to pick the cheapest fix for a failing answer: a prompt change, retrieval, structured output or a fine-tune. You get a decision table, what a fine-tune needs and never fixes, and a break-even sketch with illustrative numbers. You measure your own model's behaviour.

    At a glance

    Figure 1. From a failing answer to the cheapest fix

    flowchart TD accTitle: Decision path from a failing answer to the cheapest fix accDescr: Start with an evaluation set and a baseline. If facts are missing or change, use retrieval. Otherwise try a better prompt, examples and structured output. Only if that fails on the evaluation set, and labelled data, volume and an owner exist, fine-tune. Otherwise change the model or narrow the task. A["Answers are wrong or too costly"] --> B{"Eval set and baseline score exist?"} B -- "No" --> B1["Build them first"] B -- "Yes" --> C{"Are facts missing or changing?"} C -- "Yes" --> R["RAG: update the document, re-ingest"] C -- "No" --> D{"Do a better prompt, examples or a schema fix it on the eval set?"} D -- "Yes" --> P["Keep the prompt"] D -- "No" --> E{"Labelled examples, high volume and an owner?"} E -- "Yes" --> T["Fine-tune, then re-run the eval"] E -- "No" --> G["Change the model or narrow the task"]

    How to read this: each question is cheaper to act on than the one below it, so you reach a fine-tune only after the cheaper ones fail on your own evaluation set.

    • Measure first: without an eval set and a baseline you cannot tell whether a fix worked.
    • Facts that change belong in retrieval, not in model weights.
    • Output shape problems start with a schema and validation.
    • A fine-tune needs labelled examples, a held-out eval set and an owner.
    • It does not fix stale facts or missing access control.

    Scenario

    Composite scenario, not a real company. Lumen Industrial Supply's service desk wants emails sorted into a fixed list of categories, and a vendor says to fine-tune a model. The desk has a year of hand-sorted emails and a long prompt full of examples. Sales engineers also say the assistant sometimes quotes an old datasheet limit. The FDE must say which problem a fine-tune would fix, which it would not, and what to measure first. All numbers are illustrative.

    What you will be able to do

    • Match a failing answer to a prompt, RAG, structured output or a fine-tune.
    • List what a fine-tune needs and name two problems it does not fix.
    • Compute a break-even volume from your own numbers.
    • Prepare a chat-style training file whose splits hold no golden-set question.

    Prerequisites

    Core ideas

    Match the problem to the cheapest fix

    Figure 2. The four levers and where they sit in a request

    A request passes the Orchestrator where the prompt is built, the Retriever where facts are added, the Model gateway where the schema is enforced and the LLM provider where a fine-tuned model would runOrchestrator to Retriever (question); Retriever to Document store (lookup); Retriever to Model gateway (prompt + facts); Model gateway to LLM provider (request); Model gateway to Eval service (scored)Orchestrator: lever 1: promptOrchestratorlever 1: promptRetriever: lever 2: factsRetrieverlever 2: factsModel gateway: lever 3: schemaModel gatewaylever 3: schemaLLM provider: lever 4: tuningLLM providerlever 4: tuningDocument store: current sourcesDocument storecurrent sourcesEval service: held-out setEval serviceheld-out setquestionlookupprompt + factsrequestscored

    How to read this: levers 1 to 3 change what you send or check in your own code; lever 4 changes the model, so it comes last.

    Problem Try first Fine-tune only if
    Facts change (datasheets, prices) RAG: update the document, re-ingest never as the main fix
    Output must have a fixed shape Structured output plus validation schema and prompt still fail on your eval set
    House style or tone Prompt with a few examples examples no longer fit the prompt
    Closed vocabulary (fixed labels) Prompt with the label list, output checked against it the prompt is below your bar and you have labelled data
    Cost or latency at high volume Shorter prompt, prompt caching, another model you pass break-even
    Different data per caller Retrieval filter by caller never

    How to read this: find the failure, do the middle cell, and measure before reading the right cell.

    Anthropic's prompt engineering overview notes that latency and cost can sometimes be improved more easily by choosing a different model (as of 2026-11-04).

    What a fine-tune needs

    Anthropic's glossary defines fine-tuning as further training a pretrained model on extra data, so it mimics that dataset's patterns.

    Need What it means Owner
    Labelled examples Prompt and correct answer pairs, checked by an expert Lumen's service desk
    Eval set kept apart Examples training never sees FDE with the customer
    A baseline The prompt-only score on that set FDE
    Retraining plan A new run and eval when the model or policy changes A named owner
    Permission for the data Emails can contain personal data Customer's privacy team

    Provider facts as of 2026-11-04; check the source before relying on them:

    • OpenAI's guide says to set up evals first and hold data out as a control group, with 10 examples as the minimum and 50 as a suggested start.
    • It also says the platform is being wound down and is closed to new users.
    • Anthropic's glossary says the Claude API does not currently offer fine-tuning.

    So the provider decides whether a route exists and for how long.

    What a fine-tune does not fix

    Predict first. Lumen fine-tunes a model on its price lists, including partner-only prices. A public caller asks for a partner price. Does the fine-tune stop the leak? Decide, then open the answer.

    Answer

    No. The weights are the same for every caller and know nothing about who is asking. Access control must run in the retrieval filter, and partner-only data should stay out of the training set. The same limit covers stale facts: a changed datasheet reaches the weights only in a new run.

    Worked example

    A break-even sketch for the service desk. Assumptions: all numbers are illustrative; the price is a placeholder (use your provider's published price); any extra fee for a tuned model is ignored.

    • 300 training and 100 eval examples, 3 minutes each, at 30 per hour; 40 relabelled per refresh, 4 refreshes a year.
    • Prompt shrinks from 1,800 to 300 tokens; placeholder price 1.00 per million input tokens.

    Verified: capstone/fde-product-discovery-copilot/lessons/M3-L10/m3_l10_breakeven.py

    def year_one_cost(i: Inputs) -> tuple[float, float, float]:
    labelling = labelling_hours(i) * i.hourly_rate
    refresh = i.refresh_n * i.minutes_each / 60 * i.hourly_rate * i.refreshes_per_year
    return labelling, refresh, labelling + refresh


def saved_per_request(i: Inputs) -> float:
    return (i.tokens_before - i.tokens_after) * i.price_per_million / 1_000_000


def break_even_requests_per_month(i: Inputs) -> float:
    return year_one_cost(i)[2] / (saved_per_request(i) * 12)

    Run python lessons/M3-L10/m3_l10_breakeven.py in the capstone folder; this is its real output, and a test pins every number.

    labelling hours      20
labelling cost       600.00
refresh cost / year  240.00
year-one cost        840.00
saved per request    0.0015
break-even requests per month  46667
   20,000 requests/month  net over 12 months  -480.00
  100,000 requests/month  net over 12 months  960.00
1,000,000 requests/month  net over 12 months  17,160.00

    At 20,000 requests a month the sketch loses money; at 100,000 it gains.

    At the customer: "Our vendor says we should fine-tune." A good answer asks which failure it fixes, shows the prompt-only baseline on a held-out set, and brings a break-even sketch with the customer's numbers.

    Practice task

    The data step is worth running even if you never fine-tune: it builds the labelled set and eval set the decision needs. The sample file holds 21 reviewed pairs from four public Lumen FAQs and 6 planted defects. The script rejects a pair for a named reason and keeps golden-set questions out of every split.

    Verified: capstone/fde-product-discovery-copilot/lessons/M3-L10/m3_l10_dataset.py

    def find_leak(question: str, golden: dict[str, str], near: float = 0.6) -> str | None:
    """Return a reason naming the golden question this one repeats or nearly repeats, else None."""
    for gid, gq in golden.items():
        if normalise(question) == normalise(gq):
            return f"golden leak: same as {gid}"
        if similarity(question, gq) >= near:
            return f"golden leak: near-duplicate of {gid}"
    return None

    Run it from capstone/fde-product-discovery-copilot (it reads the 36 questions in data/eval/golden_set.json):

    ./.venv/Scripts/python lessons/M3-L10/m3_l10_dataset.py lessons/M3-L10/sample_pairs.jsonl --out "$TEMP/m3_l10_out"
    read 27, kept 21, rejected 6
  train      17
  validation 2
  test       2
  rejected line 22: duplicate question: How do I update gateway firmware?
  rejected line 23: empty: What is the standard warranty?
  rejected line 24: too long (over 200 words): What is the retention setting for a new historian?
  rejected line 25: not reviewed: How do I pair a sensor with a gateway?
  rejected line 26: golden leak: same as q20: How long does the LUM-VS-200 battery last?
  rejected line 27: golden leak: near-duplicate of q13: What is the ingress protection rating of LUM-VS-10

    Each example is one JSON Lines record, {"messages": [system, user, assistant]}. Length is counted in words, not tokens. Overlap is the share of shared words (0.6 or more is a near-duplicate), so a true paraphrase of a golden question passes (a test documents this limit): review near misses by hand.

    As of 2026-11-04, OpenAI's guide says:

    • the training file has one complete JSON structure on every line, in the chat completions format (user and assistant messages);
    • the file needs at least 10 lines.

    Check your provider's page before you convert. No training run was made here; it is left to you.

    Fill in this record for the service desk case with your own numbers.

    Row Your entry
    Pairs read, kept, rejected, and why
    Eval set size and source
    Prompt-only baseline score
    Score that is good enough
    Labelling hours, tokens saved, requests per month
    Provider price page and date read
    Tuning offered for your model today?
    Owner for retraining

    Decide first, then reveal.

    Which fix fits this Lumen case? (interactive when JavaScript is on)

    A datasheet limit changes on Monday. What do you do?

    • Start a new fine-tuning run
    • Update the document and re-ingest
    • Add the limit to every prompt
    Answer

    Update the document and re-ingest. Changing facts belong in retrieval. A fine-tune would need a new run.

    A fine-tuned model is scored on its own training examples and looks great. What is missing?

    • A longer prompt
    • A bigger training set
    • An eval set it never saw
    Answer

    An eval set it never saw. A kept-apart eval set is the control group; without it the score shows memory.

    Volume is 20,000 requests a month; break-even is 46,667. Next step?

    • Keep the prompt and revisit as volume grows
    • Fine-tune because the vendor recommends it
    • Fine-tune and skip the eval set
    Answer

    Keep the prompt and revisit as volume grows. At 20,000 the sketch nets -480 over 12 months. Skipping the eval set removes the way to judge quality.

    Exercise

    For each situation, choose the first thing to try and justify it in two sentences.

    Decision exercise.

    • A. After a datasheet revision, replies still quote the old limit.
    • B. Quote summaries need the same five headings, but some replies miss one.
    • C. The service desk sorts emails into 14 fixed categories (illustrative) with a 12-example prompt, high volume and a year of sorted emails.
    Model answer

    A: RAG; update the document and re-ingest. B: structured output with validation, so a reply missing a heading fails the check. C: score the prompt on an eval set first; only if it is below the bar and the break-even sketch passes is a fine-tune a candidate.

    Acceptance criteria: each answer names the eval set it relies on and what would change your mind.

    Done when you can answer all three without the flowchart and name one thing to measure for C.

    Hint 1: start from the failure

    Ask whether a fact, a shape or a label is wrong.

    Hint 2: for C, compute before choosing

    Run the break-even formula with a monthly volume you assume for Lumen, since the case gives none.

    Common mistakes

    • Starting a fine-tune before there is an eval set and a prompt-only baseline.
    • Fine-tuning to teach facts that change, such as prices.
    • Training on data some callers must not see.
    • Scoring the tuned model on its own training examples.
    • Letting a golden-set question, or a copy of one, into the training data.

    Knowledge check

    1. Why try retrieval before a fine-tune for a datasheet question?

    Retrieval returns the new document after re-ingest; a fine-tune needs a new run.

    2. What is the eval set for, and why must it be kept apart?

    It shows whether the change helped on unseen examples; scoring on training examples shows memory.

    3. Why can a fine-tune not enforce who may see partner prices?

    The weights are shared by all callers and know nothing about identity. The retrieval filter must run before the model sees the text.

    4. What do you do if the provider closes its tuning route?

    The provider decides which models can be tuned, so keep the labelled data and eval set, which work with any route.

    5. The sketch shows a net loss at 20,000 requests a month. Does that settle the question?

    No. It counts only token savings. A quality gain on the held-out set could still justify it.

    You can now

    • Choose between a prompt, RAG, structured output and a fine-tune for a failure.
    • Explain why fine-tuning does not fix stale facts or access control.
    • Compute a break-even volume from your own numbers.
    • Run a tested dataset step with rejection reasons and a leakage check.

    Further reading

    Last verified: 2026-11-04

    Next

    Back to the core path: ingestion

    Enroll to keep reading

    Lesson 10 of 10 · Forward Deployed Engineer: RAG, Search and Data

    Choose between a better prompt, RAG, structured output and fine-tuning with a decision table, a break-even sketch and three Lumen cases the reader decides.

    In the rest of this lesson

    • Practice task
    • Exercise
    • Common mistakes
    • Knowledge check
    • You can now
    • Further reading

    Enrolling is free. It unlocks every lesson in Forward Deployed Engineer: RAG, Search and Data, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.

    EnrollSign in

    Ask about this article

    Get answers grounded in this post. AI-generated — based on this article, and may be imperfect.

    Forward Deployed Engineer: RAG, Search and Data· Page 10 of 10
    PreviousKeep RAG Data Current: Sync, Deletes, Messy Files
    Scaled AI Weekly

    Enjoyed this? Get more like it every Monday.

    Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.

    Get new articles by email

    Free. We email a confirmation link first, and you can unsubscribe from any email.

    Join engineers building production AI systems

    Comments

    Back to all guides