Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Governance and responsible AI in practice. It is the human side of the confident-wrong RAG case, which covers measurement and abstention.
This page is about oversight design for an internal procedure-lookup assistant in an invented scenario. It is not medical, clinical or regulatory advice, and it is not guidance for patient care or for choosing a clinical system.
Not legal advice. This page shows how engineers can read the text of Regulation (EU) 2024/1689 (Official Journal, 12 July 2024, ELI), checked on 2026-10-22. It states no application dates. The Commission's AI Act page, shown as last updated 19 October 2026, describes later changes under an "AI Omnibus". Quotes below are the text as first published. Check the current text and that page, and ask counsel, before relying on any scope or date.
Context
- A hospital group's ward staff ask an assistant about internal procedures: specimen labelling, patient-transfer paperwork, equipment cleaning logs. It cites the passages it used and gives no diagnosis or dosing advice.
- About 600 questions a day. A three-week audit sampled 400 answers.
- Result: 24 were wrong (6%); staff opened the cited source for 62 (15.5%); staff acted on 21 of the 24 wrong answers (87.5%) without opening it.
| Question | What the text says | Reading |
|---|---|---|
| Is it in an Annex III area? | Point 5(a): "AI systems intended to be used by public authorities or on behalf of public authorities to evaluate the eligibility of natural persons for essential public assistance benefits and services, including healthcare services". Point 5(d): "emergency healthcare patient triage systems" | Not on these facts: it looks up internal procedures. It would change if it triaged or evaluated eligibility |
| Another route? | Article 6(1) covers AI "covered by the Union harmonisation legislation listed in Annex I" | Open: whether any product law applies to a hospital's lookup tool is for its regulatory staff |
| Why use Article 14 anyway? | Article 14(4)(b): oversight staff are to be enabled "to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)" | Not required here as far as these facts show; borrowed as good practice, since drift toward triage is plausible |
Problem statement
| Measure | Value |
|---|---|
| Wrong answers in 400 sampled | 24 (6%) |
| Source opened before acting | 62 of 400 (15.5%) |
| Wrong answers acted on without opening the source | 21 of 24 (87.5%) |
| Target | At most 6 of 24 (25%) acted on unchecked; a one-tap report; a tested stop switch |
Constraints
- Fixed: the assistant stays a procedure lookup; no clinical advice.
- Fixed: answers must arrive in seconds; a ward will not wait.
- Limit: procedure owners can review about 30 sessions a week.
- Limit: no live tests with deliberately wrong answers on real wards.
Why the traditional approach fell short
Paper binders and intranet search go stale and give no record of what was read. A "human in the loop" confirm button looks like oversight but is a click-through when the evidence is not on screen. Staff training alone does not change what a tired person does at 3 a.m.
Options considered
| Option | Verdict | Reason |
|---|---|---|
| A. Curated "current procedures" page with search (non-AI) | Kept as the fallback | Exact and cheap; the destination when the assistant is stopped or abstains |
| B. A "Confirm" button on every answer | Rejected | Staff click through without the evidence in view |
| C. Source first for high-impact topics, sampling and drills, one-tap report, stop switch | Accepted | Puts the evidence before the action and measures whether people check |
| D. Withdraw the assistant | Rejected for now | The audit points to over-reliance, which design can address; revisit if drills fail |
Solution architecture
Figure 1. Checkpoints around the answer
How to read this: the person sits at three points: before acting (source first), by sampling (owner queue) and by escalation (report); the stop switch ends the assistant safely.
Figure 2. One question in tier 1
How to read this: the opt block is the escalation path; the staff member is never required to trust the summary.
How AI solved it, step by step
Step through one question, then switch to "Without the fix".
Under the hood
Article 14(4) lists what oversight must enable. Besides the automation-bias point, it asks that people can "decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output", and "intervene in the operation of the high-risk AI system or interrupt the system through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state". The report button and the stop switch implement those two.
A drill makes the human measurable. Let a reviewer catch each of 4 seeded wrong answers with probability p, and pass at 3 or more caught. At p = 0.5 the pass probability is 0.3125; at p = 0.9 it is 0.9477. The model assumes independent catches. Placement is covered in Under the hood: Human oversight checkpoints.
What went wrong / failure modes
| Failure | Mitigation |
|---|---|
| Staff skim even with the source first | Drill scores and the report rate; change the layout when drills fail |
| Reports stay near zero | Treat it as a warning, not good news; no-blame replies from the owner |
| The tier list goes stale | Owners review it each quarter |
| Questions drift to individual patients | The Policy engine refuses patient-specific intents; reclassify before any triage feature |
| A stale procedure is retrieved | Retrieval metrics and abstention, as in the RAG case |
| The stop switch has never been tried | Quarterly test, with the curated page as the proven fallback |
Results
All numbers are illustrative targets, not measurements.
| Measure | At audit | Illustrative target |
|---|---|---|
| Wrong answers acted on unchecked | 21 of 24 (87.5%) | At most 6 of 24 (25%) |
| Tier 1 answers with the passage shown first | 0% | 100% |
| Staff passing the drill (3 of 4 caught) | Not run | At least 90% |
| Stop switch | None | Tested every quarter |
Tradeoffs and when NOT to use this approach
- Seconds, not minutes. Source-first adds a few seconds per tier 1 question.
- Not clinical governance. Anything guiding patient care needs governance beyond this page.
- Triage or eligibility. If the assistant starts to triage patients or judge eligibility, re-read Annex III point 5 and reclassify.
- A button is not oversight. Do not count a confirm click as a checkpoint.
Design rules extracted
- Put the evidence on screen before the action, and record that it was.
- Place checkpoints by impact: before the action, by sampling, by escalation.
- Measure the human: run drills and watch the report rate.
- Give staff a one-tap way to disagree and a stop switch that works.
- Do not count a confirm click as oversight.
Assumptions
All numbers are illustrative.
- Audit: 400 answers; 24 wrong = 6%; source opened 62 / 400 = 15.5%, so 338 not opened; 21 / 24 = 87.5%; target 6 / 24 = 25%.
- Drill: pass when at least 3 of 4 seeded errors are caught; P(X at least 3) = 4 x p^3 x (1 - p) + p^4, which is 0.3125 at p = 0.5 and 0.9477 at p = 0.9 (independence assumed).
- Review load: 30 sessions a week and 600 questions a day are invented, as are the hospital group, the topic tiers and the 90% pass target.
- Quotes are from the Official Journal text and the Commission page in the note.
Last verified: 2026-10-22