Skip to content
    All guides

    Forward Deployed Engineer

    Capstone Rubric (100 Points) and Submission

    Score your Product Discovery Copilot out of 100: seven weighted criteria with weak, acceptable and strong descriptions, honesty caps, and what to submit.

    By Published 8 min read

    • fde
    • capstone
    • rubric
    • evaluation
    • assessment
    Share

    This lesson is part of Forward Deployed Engineer: Evaluate and Capstone. Enroll for free, then sign in, to read it in full.

    Enroll to unlockAlready enrolled?

    What you'll learn: the seven criteria and weights that make 100 points, what weak, acceptable and strong look like for each, the caps that stop a polished page from hiding a broken build, and the exact package to submit to a reviewer or to yourself.

    How the scoring works

    Composite scenario, not a real company. The capstone is the Lumen Industrial Supply Product Discovery Copilot. Nobody here awards a certificate. The rubric is a way to find your weak spots before a customer or an interviewer does, so you can score yourself or hand the package to a peer or mentor you trust.

    Each criterion has a weight. Within it, you place your work in one of three bands. The bands are the same share of every weight: weak is up to 40 percent, acceptable is above 40 and up to 75 percent, and strong is above 75 percent, rounded down to whole points.

    # Criterion Points
    1 Correctness and tests 20
    2 Retrieval and evidence 15
    3 Workflow safety 15
    4 Security and access control 15
    5 Evaluation honesty 15
    6 Integration design note 10
    7 Documentation and demo 10
    Total 100

    The total is 20 + 15 + 15 + 15 + 15 + 10 + 10 = 100.

    Figure 1. Where the evidence for each criterion comes from

    flowchart LR accTitle: Evidence that feeds each rubric criterion accDescr: The test run feeds correctness and tests. The citations and refusals feed retrieval and evidence. The tool workflow tests feed workflow safety. The leak count and token tests feed security and access control. The evaluation output feeds evaluation honesty. The design note and the README with the demo feed the last two criteria. T["Test run, offline"] --> C1["1 Correctness and tests - 20"] R["Citations and refusals"] --> C2["2 Retrieval and evidence - 15"] W["Workflow limit and approval tests"] --> C3["3 Workflow safety - 15"] S["Leak count and token tests"] --> C4["4 Security and access - 15"] E["eval/results output"] --> C5["5 Evaluation honesty - 15"] N["Integration design note"] --> C6["6 Design note - 10"] D["README and demo"] --> C7["7 Documentation and demo - 10"]

    How to read this: every criterion is scored from an artifact you can show, not from a claim, so if the artifact does not exist the criterion starts at zero.

    The bands in points

    Weight Weak Acceptable Strong
    20 0 to 8 9 to 15 16 to 20
    15 0 to 6 7 to 11 12 to 15
    10 0 to 4 5 to 7 8 to 10

    For example, 40 percent of 15 is 6 and 75 percent of 15 is 11.25, which rounds down to 11. So a 15-point criterion is weak at 0 to 6, acceptable at 7 to 11 and strong at 12 to 15.

    1. Correctness and tests (20)

    Band What it looks like
    Weak Tests are missing, fail, or need a key or a network. Behaviour is checked by running the app by hand.
    Acceptable Tests pass offline and cover the happy path of each phase. Malformed model output and error cases are tested for some phases only.
    Strong Tests pass offline from a clean checkout, are deterministic, and cover each phase including the failure paths: malformed model output, tool failure, invalid input, wrong role, reused idempotency key.

    Evidence: the pytest output. The reference build has 151 tests and runs in a few seconds. The count is not the score; coverage of failure paths is.

    2. Retrieval and evidence (15)

    Band What it looks like
    Weak Answers are produced from the model's memory, or citations are decorative. Unsupported questions get an invented answer.
    Acceptable Chunks carry citation ids, recommendations cite them, and clearly unsupported questions are refused. Superseded or conflicting documents are not handled.
    Strong Every citation exists in the context and speaks about the product, code rejects the rest, unsupported questions are refused, superseded documents are excluded and disagreeing documents are flagged to the user.

    Evidence: the citation check, the refusal answers and the golden-set results for the refuse, superseded and conflict questions.

    3. Workflow safety (15)

    Band What it looks like
    Weak The model can loop without a limit, call any function, or trigger a sensitive action directly.
    Acceptable A step limit and a time limit exist, tools validate their arguments and are read-only, and failures are caught.
    Strong As above, plus a recorded step log, duplicate-call handling, and an approval gate for the sensitive action that binds to the exact arguments, works once, and cannot be approved by the requester.

    Evidence: the workflow and approval tests. The reference build runs three read-only tools, a limit of 14 steps by default and a 5-second default time limit, and keeps create_quote_draft behind an approval gate that is not exposed over HTTP.

    4. Security and access control (15)

    Band What it looks like
    Weak Role comes from the request body or the model, or partner-only content reaches a public user in your own tests.
    Acceptable Identity comes from a verified token and a public user is blocked from partner-only documents, but the filter runs after retrieval.
    Strong The visibility filter runs inside the retrieval query and the tools, the golden set shows zero access leaks, errors and logs contain no secrets or request text, and idempotency keys are scoped to tenant and user.

    Evidence: the access-leak column in your evaluation and the security tests. The reference build shows 5 leaks for the baseline strategy and 0 for strategies A and B.

    5. Evaluation honesty (15)

    Band What it looks like
    Weak Numbers are typed in by hand, there is no baseline, or the page implies the score is an accuracy estimate.
    Acceptable The harness ran, the page quotes its output, and a baseline is compared with at least one improvement.
    Strong Numbers are copied from the file the harness wrote, three strategies are compared, the known misses are listed and explained, and the page states that the golden set and the pipeline were written by one author and tuned together, so the numbers are in-sample regression checks.

    Evidence: eval/results/latest.md and the page that quotes it, side by side. See the evaluation report lesson.

    6. Integration design note (10)

    Band What it looks like
    Weak No note, or a restatement of the README.
    Acceptable One to two pages saying where the service sits, how identity is passed, and what happens on timeout or failure.
    Strong Acceptable, plus the sync-or-async decision with reasons, a data-refresh plan, a rollout path, and a clear statement of when a Java service would be the better home for the integration.

    Evidence: the note itself. The reference build ships no integration note, so this criterion is yours to write; the method is in the integration design note lesson.

    7. Documentation and demo (10)

    Band What it looks like
    Weak A README that does not run from a clean checkout, or a demo that shows only a success.
    Acceptable The README runs, and the demo shows a recommendation, a refusal and the evaluation table.
    Strong The README lists the limits (mock provider, in-sample evaluation, an unrun real-provider adapter), and the rehearsed demo follows the script on the demo page and includes a failure you did not hide.

    Evidence: the README and a timed run of the demo.

    Honesty caps

    Four caps stop a good-looking submission from hiding a bad build. They apply to the band, not to a penalty.

    If this is true Then this criterion cannot leave the weak band
    Your tests need a key or a network to pass 1 Correctness and tests (at most 8)
    A public user receives partner-only content in your own tests or evaluation 4 Security and access control (at most 6)
    A number on your page differs from the harness output 5 Evaluation honesty (at most 6)
    You claim a real provider works without having run it 7 Documentation and demo (at most 4)
    WARNING

    The last cap matters for the reference build too. Its Anthropic adapter has never been run against the live API. Say that, or run it with your own key and say what you saw.

    What to submit

    1. The code folder, with README.md, requirements.txt and tests.
    2. The test output, copied from a run on a clean environment, with the date.
    3. eval/results/latest.json and latest.md, exactly as the harness wrote them.
    4. A one-page evaluation summary that quotes those files and lists the known misses.
    5. The integration design note, one to two pages.
    6. The demo: either a recording of at most six minutes or the notes you will present live.
    7. A completed scoring sheet like the one below.

    A worked scoring sheet

    Assumptions: an illustrative submission, not a real person's result. The reviewer places each criterion in a band, then picks a whole number inside it.

    Criterion Band Points Out of
    1 Correctness and tests strong 17 20
    2 Retrieval and evidence acceptable 11 15
    3 Workflow safety strong 13 15
    4 Security and access control strong 14 15
    5 Evaluation honesty acceptable 11 15
    6 Integration design note acceptable 6 10
    7 Documentation and demo strong 9 10
    Total 81 100

    Each score sits inside its band from the table above: 17 is in 16 to 20, 11 is in 7 to 11, 13 and 14 are in 12 to 15, 6 is in 5 to 7 and 9 is in 8 to 10. Write the band first and the number second: a reviewer who wants to give 12 on a 15-point criterion must be able to say why it earns "strong".

    The sum is 17 + 11 + 13 + 14 + 11 + 6 + 9 = 81.

    Reading your total

    These readings are this course's own convention, not an industry standard.

    Total Reading
    Below 60 Rework the lowest criterion before you show this to anyone
    60 to 79 A working build with gaps you can name
    80 and above A build you can defend in a review

    Completing the capstone shows that you can build and measure such a service. It does not guarantee a job, and it is not a production-readiness certificate.

    Common ways to lose points

    • Quoting the evaluation from memory instead of from the output file.
    • Treating the mock provider as if it were a model.
    • Putting the visibility filter after retrieval, so the leak count passes by luck.
    • Writing a design note that describes the code instead of the customer's estate.
    • Hiding the known misses. A reviewer finds q36-style failures in minutes, and a miss you already explained scores better than one you did not.

    Last verified: 2026-11-02

    Next

    Capstone demo script and interview presentation

    Enroll to keep reading

    Lesson 7 of 8 · Forward Deployed Engineer: Evaluate and Capstone

    Score your Product Discovery Copilot out of 100: seven weighted criteria with weak, acceptable and strong descriptions, honesty caps, and what to submit.

    In the rest of this lesson

    • 2. Retrieval and evidence (15)
    • 3. Workflow safety (15)
    • 4. Security and access control (15)
    • 5. Evaluation honesty (15)
    • 6. Integration design note (10)
    • 7. Documentation and demo (10)

    Enrolling is free. It unlocks every lesson in Forward Deployed Engineer: Evaluate and Capstone, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.

    EnrollSign in

    Ask about this article

    Get answers grounded in this post. AI-generated — based on this article, and may be imperfect.

    Forward Deployed Engineer: Evaluate and Capstone· Page 7 of 8
    PreviousCapstone Architecture and API ContractNext Model-Graded Evaluation: Judge the Judge First
    Scaled AI Weekly

    Enjoyed this? Get more like it every Monday.

    Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.

    Get new articles by email

    Free. We email a confirmation link first, and you can unsubscribe from any email.

    Join engineers building production AI systems

    Comments

    Back to all guides