Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Governance and responsible AI in practice.
Not legal advice. This page shows how engineers can read the text of Regulation (EU) 2024/1689 (Official Journal, 12 July 2024, ELI), checked on 2026-10-22. It states no application dates. The Commission's AI Act page, shown as last updated 19 October 2026, describes later changes under an "AI Omnibus". Quotes below are the text as first published. Check the current text and that page, and ask counsel, before relying on any scope or date.
Context
- A consumer lender decides about 6,000 small personal-loan applications a month. Credit analysts make every decision.
- An assistant reads the application file and the lender's policy documents, then drafts a summary, a recommended outcome and reasons. The analyst accepts, edits or overrides.
- About 1.5% of decisions are disputed: 90 a month.
| Question | What the text says | Reading |
|---|---|---|
| Is it in an Annex III area? | Point 5(b) of Annex III: "AI systems intended to be used to evaluate the creditworthiness of natural persons or establish their credit score, with the exception of AI systems used for the purpose of detecting financial fraud" | Likely yes: the assistant recommends an outcome on creditworthiness. The lender's separate fraud screen is a different system |
| Is there room for doubt? | Article 6(3) allows an Annex III system not to be high-risk where, for example, it "is intended to improve the result of a previously completed human activity" | Open. A drafted recommendation that analysts mostly follow sits close to the decision. Default: treat it as high-risk |
Problem statement
| Measure | Value |
|---|---|
| Disputed decisions a month | 90 (1.5% of 6,000) |
| Reconstructable in an audit sample of 20 | 6 (30%) |
| What was missing | Prompt and model version, policy document version, what the analyst saw |
| Target | 20 of 20 reproduced from records within one working day |
Constraints
- Fixed: customer data is personal data; logs must not become a second copy of the customer file.
- Fixed: the team that runs the assistant must not be able to edit the records.
- Limit: retention is set by the lender's legal team and financial regulation. Article 26(6) says deployers keep the logs under their control "for a period appropriate to the intended purpose of the high-risk AI system, of at least six months, unless provided otherwise in applicable Union or national law, in particular in Union law on the protection of personal data".
- Limit: for financial institutions the same paragraph says logs are kept "as part of the documentation kept pursuant to the relevant Union financial service law".
Why the traditional approach fell short
A hand-built scorecard is auditable but cannot read free-text documents. Application logs rotate after 30 days and record request text, not which prompt, model and policy version produced it. Analyst notes are free-form and say what the analyst thought, not what the assistant showed.
Options considered
| Option | Verdict | Reason |
|---|---|---|
| A. Analyst decides from a checklist and a case file (non-AI) | Kept as the fallback and the override path | Fully explainable; slower on long files |
| B. Keep application logs, extend retention to 24 months | Rejected | Spreads raw personal data, still lacks versions, and can be edited |
| C. Append-only decision record holding ids and hashes, versioned policy documents and prompts, raw text kept in the case system | Accepted | Replays a decision without copying the customer file |
Solution architecture
Figure 1. Where the record is written
How to read this: the record stores ids and hashes; the policy text and the draft live in systems that already hold them under their own retention.
Figure 2. One decision, then a dispute
How to read this: the lower half is the audit; it works only because the upper half stored versions, not just text.
How AI solved it, step by step
Step through one decision and its dispute, then switch to "Without the fix".
Under the hood
Article 12(1) says high-risk AI systems "shall technically allow for the automatic recording of events (logs) over the lifetime of the system".
Article 12(2) asks for "a level of traceability of the functioning of a high-risk AI system that is appropriate to the intended purpose of the system". Article 86(1) gives a person affected by a deployer's decision on an Annex III system's output, where it has legal or similarly significant effects, a right to "clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken". The record must hold what that needs.
{
"decision_id": "string", "sequence": "integer", "created_at": "timestamp",
"prompt_version": "id", "model_version": "id",
"policy_refs": ["document id + version id"],
"draft_sha256": "hash", "analyst_id": "id",
"action": "accept | edit | override", "reason_code": "code",
"prev_hash": "hash", "record_hash": "SHA-256(prev_hash + the fields above)"
}Each hash covers the one before, so editing an old record breaks every later hash. That detects tampering; write-once storage and removing update rights are what prevent it. Detail is in Under the hood: Audit logging that survives an audit.
What went wrong / failure modes
| Failure | Mitigation |
|---|---|
| A record is edited | Write-once storage, no update right, nightly chain check that alerts on a break |
| A policy document was changed in place | The Document store keeps versions; the record holds the version id |
| Logs hold raw personal data past their purpose | The record holds ids and hashes; raw text follows the case system's retention, set with counsel |
| Analysts stop questioning drafts | Track the override rate; review below 3%. See the healthcare oversight case |
| A gap in the sequence | The sequence number exposes a missing record |
Results
All numbers are illustrative targets, not measurements.
| Measure | Before | Illustrative target |
|---|---|---|
| Disputed decisions reconstructable | 6 of 20 (30%) | 20 of 20, within one working day |
| Record size | None kept | 12 MB a month, 288 MB over 24 months |
| Records the assistant team can edit | All logs | None |
| Override rate | Not tracked | About 9% (540 of 6,000), reviewed below 3% |
Tradeoffs and when NOT to use this approach
- A record proves what happened, not that it was right. Evaluation still decides quality.
- Low-stakes drafting. A summariser for internal notes needs ordinary logs, not a sealed chain.
- Scope. Article 27(1) names "deployers of high-risk AI systems referred to in points 5 (b) and (c) of Annex III" for a fundamental rights impact assessment; whether it reaches this lender is for counsel.
Design rules extracted
- Log versions and ids, not copies of personal data.
- Version every input that shapes a decision: prompt, model, policy text.
- Seal records so the team that runs the system cannot edit them.
- Build the customer explanation from the record, not from memory.
- Test the audit: replay a sample every week.
Assumptions
All numbers are illustrative.
- 6,000 decisions a month; 1.5% disputed = 90. Audit sample 20, of which 6 reconstructable = 30%; the other 14 had no versions or had rotated logs.
- Record about 2 KB: 6,000 x 2 KB = 12,000 KB = 12 MB a month (decimal); 12 x 24 = 288 MB over 24 months. Retention of 24 months is an example, not a legal figure.
- Override rate 9%: 0.09 x 6,000 = 540; the review trigger 3% = 180 is a team choice.
- Quotes are from the Official Journal text and the Commission page in the note. The lender and its counts are invented.
Last verified: 2026-10-22