Skip to content
    All guides

    Forward Deployed Engineer

    Build, Buy or Hybrid: AI Vendor Selection Matrix

    Choose between build, buy and hybrid for an AI capability with must-have gates, a weighted scoring matrix, a tie rule and a sensitivity check.

    By Published 7 min read

    • fde
    • solution-architect
    • pre-sales
    • build-vs-buy
    • vendor-selection
    • scoring-matrix
    Share

    This lesson is part of AI Solution Architect: Estimate, Propose and Defend. Enroll for free, then sign in, to read it in full.

    Enroll to unlockAlready enrolled?

    What you'll learn: how to compare build, buy and hybrid options so a customer can audit the result: gates, weights fixed first, weighted totals, a tie rule and a sensitivity check. Vendors here are placeholders and the scores are illustrative.

    At a glance

    Figure 1. From options to a recorded decision

    flowchart TD accTitle: Vendor selection from must-have gates to a recorded decision accDescr: List the options, then apply the must-have gates. Options that fail a gate are excluded. The rest are scored against weights fixed beforehand, totals are computed and a sensitivity check is run. A gap larger than the tie rule is recorded as a decision. A close result goes to a short trial. A["List options: build, hybrid, buy"] --> B{"Passes every must-have?"} B -- "no" --> X["Exclude, record the reason"] B -- "yes" --> C["Weights fixed before scoring"] C --> D["Score each criterion from evidence"] D --> E["Weighted totals and a sensitivity check"] E --> F{"Gap larger than the tie rule?"} F -- "yes" --> G["Record the decision"] F -- "no" --> H["Short trial on the customer's data"] H --> D

    How to read this: gates remove options, weights and scores rank the rest, and a close result sends you to evidence, not to a debate.

    • A must-have is a gate, not a score: an option that fails it is out.
    • Fix the weights before you see any scores.
    • Treat gaps smaller than one scorer's disagreement as ties.

    Scenario

    Composite scenario, not a real company. Lumen Industrial Supply's read-only Order and Policy Assistant needs an owner. The platform team prefers to build, the Head of Customer Operations wants a packaged product, and the IT security lead has two non-negotiable rules.

    Options: build; Vendor A, a packaged assistant; Vendor B, a platform Lumen extends with its own code; Vendor C. All vendors are placeholders, not real products.

    What you will be able to do

    • Write gates and weighted criteria, with weights that sum to 100.
    • Compute weighted totals and apply a tie rule.
    • Run a sensitivity check and record the decision.

    Prerequisites

    Core ideas

    What you keep responsibility for

    Option You own The vendor owns
    Build Retrieval, workflow, evaluation, operations The model service
    Hybrid Your workflow code, connectors, evaluation The platform and hosting
    Buy Configuration and usage The product and roadmap

    Assume the mindset lesson's first step, configuring what you own, already failed.

    Gates, weights and evidence

    A gate is a yes or no question with a written source; UK procurement guidance likewise separates must haves from nice to haves (Government Commercial Agency, accessed 2026-11-03). Lumen's gates are illustrative placeholders; real ones come from the platform team and the IT security lead.

    • G1: no new public endpoint, so traffic enters through the existing API gateway with the agent's identity (owner: platform team).
    • G2: logs hold no customer personal data (owner: IT security lead).

    Vendor C cannot confirm G1 in writing, so it is excluded.

    Criterion Evidence behind a score Weight
    Constraint fit Written answers on security and platform constraints beyond G1 and G2 25
    Quality on test set Trial pass rate, including the old-policy trap 25
    Integration effort A short spike on order lookup 15
    Three-year cost A TCO table like the estimate 15
    Exit and portability Data and prompts exportable 10
    Continuity risk Roadmap, team skills 10

    Scores run from 1 (poor) to 5 (strong). Weighted total = sum of weight times score, divided by 5, so a perfect option scores 100.

    In the interview: "How would you decide between building and buying?" A strong answer asks for must-haves first, agrees weights, scores from a trial rather than a demo, and says what would change the decision.

    WARNING

    Never change weights after seeing the result unless you record why.

    Worked example

    Assumptions: Build means a thin Python service plus a hosted model. The scores are illustrative stand-ins for evidence, chosen so the example reaches the project's decision. Real scores are your own judgement from trial results and may differ; the method, not these numbers, is the lesson.

    Criterion Weight Build Vendor A Vendor B
    Constraint fit 25 5 3 4
    Quality on test set 25 4 3 4
    Integration effort 15 3 4 3
    Three-year cost 15 4 3 3
    Exit and portability 10 5 2 4
    Continuity risk 10 3 2 3
    Weighted total 100 82 59 72

    Build: (125 + 100 + 45 + 60 + 50 + 30) / 5 = 410 / 5 = 82. Vendor A: 295 / 5 = 59. Vendor B: 360 / 5 = 72.

    Tie rule: one step on a 25-weight criterion moves a total by 25 / 5 = 5 points, so a gap of 5 or less is one scorer's disagreement. Build leads Vendor B by 10: not a tie. Change a weight or a score below and see whether the leader and the verdict change.

    Weighted decision matrix (interactive when JavaScript is on)
    Build82 (leader)
    Vendor A59
    Vendor B72
    Gap between the top two10 points
    Tie rule: one step on the heaviest criterion5 points
    VerdictBuild leads Vendor B by 10: not a tie

    Figure 2. The decision: what Lumen builds, reuses and buys

    Build, reuse and buy for the Order and Policy AssistantAPI gateway to Orchestrator (agent identity); Orchestrator to Retriever (policy query); Orchestrator to Order system (reads order); Orchestrator to Model gateway (prompts); Model gateway to LLM provider (calls)API gateway: existing, reusedAPI gatewayexisting, reusedOrchestrator: thin, builtOrchestratorthin, builtRetriever: keyword, builtRetrieverkeyword, builtOrder system: existing, reusedOrder systemexisting, reusedModel gateway: configuredModel gatewayconfiguredLLM provider: hosted, boughtLLM providerhosted, boughtagent identitypolicy queryreads orderpromptscalls

    How to read this: rust-outlined boxes are built, green boxes exist already, grey boxes are bought or configured.

    Sensitivity check, weights in table order:

    Weights Build Vendor A Vendor B
    Security-led: 40, 10, 15, 15, 10, 10 85 59 72
    Speed-led: 10, 15, 40, 15, 10, 10 74 64 67
    Quality-led: 15, 40, 15, 15, 5, 10 79 60 72

    Build leads under all three, but its lead of 7 under speed-led and quality-led is less than one step on a 40-weight criterion (8 points), so those two are close calls: check the evidence behind the heavy criterion. Record the decision: a thin orchestration service behind the existing gateway, a hosted model through a model gateway, keyword search first, and a read-only minimum viable product (MVP). It is a hybrid in practice: build the thin layer, buy the model.

    Practice task

    Add your decision matrix to the design pack, then test how far the decision can slip. A spike shows Build's quality is 3, not 4, and its integration score is 2, not 3.

    1. Recompute Build and compare it with Vendor B (72) using the tie rule.
    2. Write a three-line decision record: decision, evidence, what would change it (AI Architect templates).

    Check: Build is (125 + 75 + 30 + 60 + 50 + 30) / 5 = 74, a lead of 2: a tie. The record says how it was broken, for example a trial of Vendor B on the test set.

    Exercise

    Decision exercise. A trial shows Vendor B writes customer names into its logs; switching this off is "on the roadmap". Options: (A) keep it, constraint-fit score lowered to 2; (B) exclude it, as it fails G2; (C) keep it and ask for a written date. Choose and justify.

    Done when your answer names the gate and says whether the total matters.

    Hint 1: gate or score?

    Is G2 a rule or a preference?

    Hint 2: what would reopen it?

    A roadmap item is a claim. What evidence would you need?

    Model answer

    Option B. G2 is a gate, so the vendor is out whatever its total; lowering a score (A) lets other criteria outvote a rule the security lead set. Record the date and evidence, and reopen only if the setting ships and a trial confirms clean logs. C is a note in that record, not a reason to keep the vendor.

    Common mistakes

    • Treating a must-have as a score.
    • Choosing weights after seeing the scores.
    • Scoring quality from a demo, not a trial on the customer's questions.
    • Reporting decimals; scores are judgements.

    Knowledge check

    1. Why are must-haves gates instead of heavily weighted criteria?

    A weighted criterion can be outvoted, so an option that breaks a hard rule might still win. A gate removes it first, because no total makes a prohibited option acceptable.

    2. How is Vendor B's total of 72 computed?

    Add weight times score: 100 + 100 + 45 + 45 + 40 + 30 = 360. Dividing by the top score of 5 makes a perfect option score 100.

    3. Why is a gap of 2 a tie but a gap of 10 not?

    One step on a 25-weight criterion moves a total by 5 points, and scorers often differ by a step. A lead of 2 can vanish; a lead of 10 needs two such disagreements.

    4. What is the cheapest way to break a tie?

    Measure the claim behind the largest weighted difference in a short trial on the customer's data. A measurement replaces an opinion at far less cost than a wrong three-year commitment.

    5. Why fix the weights before scoring?

    Weights changed after you see totals can steer the result to a favourite, consciously or not. Agreeing them first keeps it defensible.

    You can now

    • Add the build, buy or hybrid decision matrix to your design pack.
    • Write gates and weights, and compute totals.
    • Apply a tie rule and a sensitivity check.

    Further reading

    Last verified: 2026-11-03

    Next

    RFP and RFI responses, and statements of work

    Enroll to keep reading

    Lesson 4 of 15 · AI Solution Architect: Estimate, Propose and Defend

    Choose between build, buy and hybrid for an AI capability with must-have gates, a weighted scoring matrix, a tie rule and a sensitivity check.

    In the rest of this lesson

    • Worked example
    • Practice task
    • Exercise
    • Common mistakes
    • Knowledge check
    • You can now

    Enrolling is free. It unlocks every lesson in AI Solution Architect: Estimate, Propose and Defend, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.

    EnrollSign in

    Ask about this article

    Get answers grounded in this post. AI-generated — based on this article, and may be imperfect.

    AI Solution Architect: Estimate, Propose and Defend· Page 4 of 15
    PreviousAI Project Estimation: Effort, Tokens and TCONext RFP, RFI and Statement of Work for AI Projects
    Scaled AI Weekly

    Enjoyed this? Get more like it every Monday.

    Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.

    Get new articles by email

    Free. We email a confirmation link first, and you can unsubscribe from any email.

    Join engineers building production AI systems

    Comments

    Back to all guides