Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
Context
The in-house legal department of a mid-sized company has 40 lawyers and paralegals and about 18,000 contracts in its document management system, with roughly 3,000 more added each year. Lawyers ask questions that cut across contracts: which supplier agreements allow termination for convenience on under 90 days' notice, which need consent on a change of control. Today they combine folders, keyword search and paralegals who read contracts.
An engineer built a retrieval-augmented prototype in a few weeks, and the department asked for a plan to take it into production.
Problem statement
The build plan estimated 198k currency units over three years: 144k to build (two engineers for six months) and 18k a year to run. Five months in, a costing exercise that included maintenance, evaluation and legal review of outputs projected 438k, which is 121% above the plan, and the work was already underway. Nobody had compared the build with buying or with a simpler option; the prototype had made building feel like the default.
Measurable goal: choose the approach on a three-year total cost and on a pass rate against 100 real lawyer questions, before more build effort is spent.
Constraints
- Fixed: contracts are confidential; only providers on the approved list may process them, and every answer must link to the clause it came from.
- Fixed: access follows matter-level permissions; some contracts are restricted to named lawyers.
- Fixed: a lawyer is accountable for every conclusion; the tool finds clauses.
- Negotiable: the delivery date and the clause types covered first.
- Limit: two engineers part time, none with search-service experience.
Why the traditional approach fell short
Keyword search misses wording variants: "terminate for convenience", "terminate without cause" and "exit on notice" mean similar things and share few words. It also returns whole documents, not the clause. Metadata and folders only hold what someone typed at intake (counterparty, date, value), so clause-level attributes are not there. The manual process is accurate but slow: reading the 4,000 in-force contracts for one question at 15 minutes each takes 1,000 hours.
Options considered
| Option | Verdict | Reason |
|---|---|---|
| A. Configure the existing system (non-AI): intake fields, paralegal tagging of 12 common clause types on in-force contracts, saved searches | Baseline, kept as fallback | Cheapest, no new vendor, explainable; answers only the 12 tagged types |
| B. Buy a contract-search product | Preferred, subject to a trial | Handles free-form questions; vendor shares maintenance; trial must pass on the department's own contracts |
| C. Build retrieval over the contracts | Rejected for now | Highest three-year cost; the department would own evaluation, permissions and maintenance |
The ADR carries a review trigger: build is reconsidered only if the trial of option B fails a fixed constraint or the annual licence exceeds the break-even under "Under the hood".
Solution architecture
Figure 1. Contract search components with matter-level access
How to read this: the policy engine filters by matter before search, and every query leaves a trace for audit and for the monthly evaluation.
Figure 2. One question, with the no-answer path
How to read this: the answer is built only from clauses the user may see, and an empty result is reported honestly instead of guessed.
How AI solved it, step by step
- Ask. A lawyer asks, for example, which supplier contracts allow termination for convenience on under 90 days' notice.
- Authenticate. The gateway identifies the user through single sign-on.
- Filter. The policy engine returns the matters this user may see. Decision: restricted contracts are excluded before any search happens.
- Retrieve. The retriever ranks clause-level passages in permitted contracts, by meaning and by keyword.
- Threshold. If no passage scores above the agreed threshold, the system says so. Decision: no answer is better than an unsupported one.
- Draft. The model gateway asks an approved model for a two-sentence answer using only those passages.
- Cite. The answer links to each clause, so the lawyer checks the source.
- Trace. The query, results, user and versions are stored for audit.
- Evaluate. Each month the evaluation service reruns the 100-question set against the requirement.
Under the hood
The key step is the cost model, because the failure in this case was arithmetic. The plan counted the build and the running cost. The projection added the lines it left out: engineers keeping the system working (half an engineer a year) and evaluation with legal review of outputs.
Break-even is the licence price at which buying costs the same as building. Buy costs 3 x licence plus 141.4k of non-licence costs (implementation 40k, vendor review 15k, three years of internal administration at 28.8k a year, which is 86.4k). Setting that equal to the 438k build projection gives licence = (438 - 141.4) / 3 = 98.9k a year. Below that figure buying is cheaper than building on these assumptions; above it the answer flips, and the ADR's review trigger watches that figure. The method to write the requirement and the ADR is in AI Architect mindset: requirements, ADRs and design reviews, under "Build vs buy vs configure".
What went wrong / failure modes
- Cost overrun from omitted lines. The plan priced the build, not the system's life. Mitigation: the ADR must include a three-year total with maintenance, evaluation and review lines.
- Scope growth. Scanned-document text extraction, matter permissions and citations were not in the first estimate. Mitigation: list them as requirements first and price every option against all of them.
- A demo passes and your data fails. Mitigation: trial on 100 real questions against the department's own contracts, with a stated pass mark.
- Superseded versions retrieved. An amended contract may be returned beside its replacement. Mitigation: index the current version and show effective dates.
- Permission leak. Search that ignores matter access exposes restricted contracts. Mitigation: filter by permission before retrieval and test with a case that must return nothing.
Results
These results are illustrative targets and arithmetic, not measurements. The three-year comparison, in thousands of currency units:
| Option | Year 1 | Year 2 | Year 3 | Three-year total | Free-form questions |
|---|---|---|---|---|---|
| A. Configure (non-AI) | 99.65 | 40.65 | 40.65 | 180.95 | No, 12 tagged clause types only |
| B. Buy | 143.8 | 88.8 | 88.8 | 321.4 | Yes, subject to the trial |
| C. Build (projected) | 218 | 110 | 110 | 438 | Yes, after build |
| C. Build (original plan) | n/a | n/a | n/a | 198 | Yes |
The gap between the original plan (198k) and the projected build (438k) is 240k, which is 121% of the plan. Decision rule: a candidate passes if the correct clause is in its top five results for at least 80 of 100 questions; among those that pass, choose the lowest three-year total unless a fixed constraint rules it out. Option A is expected to fail that test on questions outside its 12 tagged clause types, which is why B is preferred; if the 100 questions turn out to be mostly about those 12 types, A passes and wins on cost.
Tradeoffs and when NOT to use this approach
- Do not buy when the data cannot leave your boundary and no product runs inside it.
- Do not buy when licences pass the break-even, about 98.9k a year on these assumptions.
- Do not build for a common need unless a constraint forces it. Some teams build to control the data path and avoid lock-in; that is a legitimate tradeoff if the cost is priced and approved up front.
- Do not use AI when configure is enough. If 12 tagged clause types cover the questions, option A costs about 56% of option B and is simpler to explain.
Design rules extracted
- Price every option over three years, including maintenance, evaluation and review, before approving any build.
- Always include a simple non-AI option; it sets the baseline the AI option must beat.
- Run each option against the same real questions with a written pass mark.
- Write the break-even price into the ADR as a review trigger.
- Apply permissions as a filter before search and test it with a case that must return nothing.
Assumptions
All numbers are illustrative; k means thousand currency units. An engineer-month costs 12k fully loaded, and a paralegal hour costs 35 currency units (0.035k).
- Contracts: 18,000 in total, 4,000 in force and in scope, 3,000 added a year. Manual reading: 4,000 x 0.25 h = 1,000 hours for one question.
- Build (projected). Year 1: build 2 engineers x 6 months x 12 = 144; support 0.5 engineer x 6 months x 12 = 36; running (hosting and model usage) 18; evaluation and legal review of outputs 20; total 218. Years 2 and 3: support 0.5 x 12 x 12 = 72, running 18, evaluation 20; total 110 each. Three years: 218 + 110 + 110 = 438.
- Build (original plan). 144 + 3 x 18 = 198. Overrun: (438 - 198) / 198 = 1.21 (121%).
- Buy. Licence 60 a year; implementation and migration 40 once; vendor security review 15 once; internal administration 0.2 engineer x 12 months x 12 = 28.8 a year. Year 1: 60 + 40 + 15 + 28.8 = 143.8. Years 2 and 3: 60 + 28.8 = 88.8 each. Total 143.8 + 88.8 + 88.8 = 321.4.
- Configure. Setup 2 engineer-months x 12 = 24; initial tagging 4,000 contracts x 0.25 h x 35 = 35; ongoing tagging 3,000 x 0.25 h x 35 = 26.25 a year; administration 0.1 engineer x 12 x 12 = 14.4 a year. Year 1: 24 + 35 + 26.25 + 14.4 = 99.65. Years 2 and 3: 26.25 + 14.4 = 40.65 each. Total 99.65 + 40.65 + 40.65 = 180.95.
- Break-even licence. (438 - (40 + 15 + 86.4)) / 3 = (438 - 141.4) / 3 = 98.87 a year.
- Ratio of configure to buy: 180.95 / 321.4 = 0.563 (about 56%).
- Trial: 100 lawyer questions, pass mark 80 with the correct clause in the top five results, and a 15-minute read per contract. These are chosen for illustration, not measured.
Last verified: 2026-10-21