Skip to content

    AI Architect

    Governance and Responsible AI in Practice

    How AI architects turn governance into engineering: an AI system registry, EU AI Act risk tiers, audit logs, data lineage, system cards and human oversight.

    By Published 28 min read

    • ai-architect
    • architecture
    • governance
    • eu-ai-act
    • audit
    • responsible-ai
    Share

    This lesson is part of AI Architect: From AI Engineer to Expert. Enroll for free, then sign in, to read it in full.

    What you'll learn: how to run AI governance as engineering: keep a registry of AI systems, classify each one against the EU AI Act's risk tiers using the official text, record audit evidence per decision, trace an answer back to its source document, document models and systems, and place human oversight where it works. Not legal advice; the legal text was checked on 2026-10-22.

    At a glance

    Figure 1. From inventory to evidence

    flowchart LR accTitle: From inventory to evidence accDescr: An inventory of AI systems feeds risk classification. Classification drives the system card and the controls. Controls write to a write-once audit log. Monitoring and incidents loop back to classification through change control, and the card, the log and the monitoring records form the evidence pack for an auditor. INV["AI system inventory"] --> CLS["Risk classification"] CLS --> DOC["System card and records"] CLS --> CTL["Controls: oversight, logging, disclosure"] CTL --> LOG["Audit log (write-once)"] CTL --> MON["Monitoring and incidents"] MON -- "incident or change" --> CHG["Change control"] CHG --> CLS DOC --> EVD["Evidence pack for an auditor"] LOG --> EVD MON --> EVD

    How to read this: governance is the loop on the left (list, classify, control, monitor, change); the evidence pack on the right is what that loop leaves behind.

    • Governance is a set of records your systems produce while they run, not a policy document.
    • Start with an inventory: a system that is not listed cannot be classified, monitored or switched off.
    • The risk tier follows the intended use, not the model, so record the intended purpose per system.
    • Log what lets a reviewer rebuild a decision, in storage the application team cannot edit, for no longer than its purpose needs.
    • Human oversight is a designed checkpoint with measurable behaviour, such as an override rate.
    • The law moves: this page quotes the official text as checked on 2026-10-22; check again before relying on it.

    The real-world problem

    An internal assistant has helped screen applications for six months. An auditor asks three questions: which prompt version produced last Tuesday's recommendation, who reviewed it, and which policy document it relied on. Nobody can answer without a week of digging. Governance makes them a query.

    After this page you will be able to

    • Write a registry record for an AI system with the fields an auditor asks for.
    • Classify five scenarios against Article 6 and Annex III using the quoted text, and recognise when the honest answer is "ask counsel".
    • Design a per-decision audit record, a retention rule and a tamper-evident store.
    • Trace an answer from a source document version to the person who approved it.
    • Place oversight checkpoints and measure whether they work.

    Terms: AI system inventory, intended purpose, high-risk AI system, audit trail, data lineage, model card, system card and automation bias.

    WARNING

    Not legal advice. Date checked: 2026-10-22. This page quotes the official text of Regulation (EU) 2024/1689 and says what official EU pages said on that date. What applies to your organisation depends on your role, each system's intended purpose and national law. Ask qualified counsel, and check the official pages again: the text and its dates have been amended and may change again.

    Core concepts

    A registry of AI systems

    Figure 2. The registry: one system, many versions, many decisions

    erDiagram accTitle: AI system registry accDescr: One AI system has one risk record, many versions and many incidents. Each version has one system card, many evaluation runs and produces many decision log entries. AI_SYSTEM ||--|| RISK_RECORD : "classified by" AI_SYSTEM ||--o{ SYSTEM_VERSION : "has" AI_SYSTEM ||--o{ INCIDENT : "has" SYSTEM_VERSION ||--|| SYSTEM_CARD : "described by" SYSTEM_VERSION ||--o{ EVAL_RUN : "tested by" SYSTEM_VERSION ||--o{ DECISION_LOG : "produces" AI_SYSTEM { string system_id PK string owner string intended_purpose string role } SYSTEM_VERSION { string version_id PK string model_id string prompt_version string retrieval_config }

    How to read this: the risk record hangs off the system because the intended purpose does; everything that changes with a release hangs off the version.

    Field What an auditor asks Illustrative value
    System id, owner, status Who answers for it? hr-screening-assistant, platform team, live
    Intended purpose and context What was it built to do, for whom? "Rank applications for recruiter review"
    Role Provider, deployer or both? Decided by counsel, dated
    Risk record Which tier, who decided, why, when? Annex III point 4(a), treated as high-risk
    Versions Which model, prompt, retrieval setup were live when? model id, prompt p-14, index i-9
    Evidence pointers Where are the evaluation runs, card, oversight design? ids that resolve to files
    Incidents and changes What went wrong or changed? ids, dates, outcome

    Article 3(12) defines intended purpose, and Article 3(3) and 3(4) define provider and deployer. A team that wraps a third-party model in its own assistant may be either or both: a legal question, so the registry records the answer and its date.

    Regulation (EU) 2024/1689, Article 3(12). ‘intended purpose’ means the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation;

    Check yourself: two teams call the same model API. One builds a canteen menu assistant, the other an application filter. Which registry field makes their obligations differ?

    The intended purpose and context of use. Article 3(12) ties intended purpose to the specific context, and Annex III (next section) lists uses, not models.

    Risk tiers and the text that defines them

    Figure 3. Reading the classification text (a study aid, not the legal test)

    flowchart TD accTitle: Reading the EU AI Act classification text accDescr: A system with an intended purpose is first checked against Article 6(1) for products under Annex I law. If not, it is checked against the Annex III areas. An Annex III system that performs profiling is high-risk. Otherwise the Article 6(3) conditions are tested and, if met, the assessment is documented. A separate Article 50 check applies to systems that talk to people or generate content. S["System and its intended purpose"] --> A1{"Article 6(1): Annex I product needing third-party conformity assessment?"} A1 -- yes --> H1["High-risk (Article 6(1))"] A1 -- no --> A3{"Intended use in an Annex III area? (Article 6(2))"} A3 -- no --> T{"Talks to people or generates content? (Article 50)"} A3 -- yes --> P{"Profiling of natural persons?"} P -- yes --> H2["High-risk, no Article 6(3) exception"] P -- no --> C3{"An Article 6(3) condition met?"} C3 -- no --> H3["High-risk (Article 6(2))"] C3 -- yes --> D["Not high-risk, assessment documented (Article 6(4))"] T -- yes --> TR["Disclosure duties (Article 50)"] T -- no --> O["No trigger in the text quoted here"]

    How to read this: follow the arrows from the intended purpose; the Article 50 check is separate and can also apply to a high-risk system (drawn only after the Annex III test).

    Article 6(2). In addition to the high-risk AI systems referred to in paragraph 1, AI systems referred to in Annex III shall be considered to be high-risk.

    Article 6(3). By derogation from paragraph 2, an AI system referred to in Annex III shall not be considered to be high-risk where it does not pose a significant risk of harm to the health, safety or fundamental rights of natural persons, including by not materially influencing the outcome of decision making. The first subparagraph shall apply where any of the following conditions is fulfilled: (a) the AI system is intended to perform a narrow procedural task; (b) the AI system is intended to improve the result of a previously completed human activity; (c) the AI system is intended to detect decision-making patterns or deviations from prior decision-making patterns and is not meant to replace or influence the previously completed human assessment, without proper human review; or (d) the AI system is intended to perform a preparatory task to an assessment relevant for the purposes of the use cases listed in Annex III. Notwithstanding the first subparagraph, an AI system referred to in Annex III shall always be considered to be high-risk where the AI system performs profiling of natural persons.

    Article 6(4). A provider who considers that an AI system referred to in Annex III is not high-risk shall document its assessment before that system is placed on the market or put into service.

    Figure 4. The eight Annex III areas, with the points the quiz uses

    Point Area (exact heading in Annex III) Wording used below
    1 Biometrics, in so far as their use is permitted under relevant Union or national law -
    2 Critical infrastructure -
    3 Education and vocational training -
    4 Employment, workers’ management and access to self-employment 4(a): "...the recruitment or selection of natural persons, in particular to place targeted job advertisements, to analyse and filter job applications, and to evaluate candidates"
    5 Access to and enjoyment of essential private services and essential public services and benefits 5(b): "...to evaluate the creditworthiness of natural persons or establish their credit score, with the exception of AI systems used for the purpose of detecting financial fraud"
    6 Law enforcement, in so far as their use is permitted under relevant Union or national law -
    7 Migration, asylum and border control management, in so far as their use is permitted under relevant Union or national law -
    8 Administration of justice and democratic processes -

    How to read this: Annex III lists uses ("intended to be used for ..."), grouped into eight areas; point 5 includes creditworthiness with its own fraud-detection exception.

    Classification attaches to an intended use, so one model can sit in different tiers in different products. Article 6(3) is a set of conditions, not a free pass, and the text requires a documented assessment. Where the conditions are only partly met, the answer is a documented decision with counsel.

    Try the quiz before opening each answer. Item 5 is deliberately ambiguous.

    Classify the scenario using the quoted text (interactive when JavaScript is on)

    A vendor sells a tool intended to analyse and filter job applications and to evaluate candidates. Where does the text place it?

    • Annex III, employment (point 4(a)): high-risk under Article 6(2)
    • Annex III, education (point 3)
    • Outside Annex III: only Article 50 applies
    • Not covered by any provision quoted here
    Answer

    Annex III, employment (point 4(a)): high-risk under Article 6(2). Point 4(a) covers analysing and filtering job applications and evaluating candidates; Article 6(2) makes Annex III systems high-risk.

    A bank assistant is intended to evaluate the creditworthiness of natural persons who apply for a loan.

    • Annex III, essential services (point 5(b)): high-risk under Article 6(2)
    • Annex III, employment (point 4)
    • Outside Annex III because banks are not named
    • Only a disclosure duty under Article 50(1)
    Answer

    Annex III, essential services (point 5(b)): high-risk under Article 6(2). Point 5(b) lists systems 'to evaluate the creditworthiness of natural persons or establish their credit score'.

    A model is used only to detect financial fraud on card transactions. Does point 5(b) make it high-risk?

    • Yes: any AI system a bank uses is listed
    • No under point 5(b): its text excepts systems used for detecting financial fraud
    • Yes, because fraud scores affect customers
    • Only if the bank is a public authority
    Answer

    No under point 5(b): its text excepts systems used for detecting financial fraud. Point 5(b) ends 'with the exception of AI systems used for the purpose of detecting financial fraud'. Another point or Article 6(1) is a separate question.

    A retail website chatbot answers delivery questions. No Annex III area fits. What does the quoted text still require?

    • Nothing: only high-risk systems have duties
    • Article 50(1): tell people they are interacting with an AI system, unless that is obvious
    • Article 12 logging
    • Article 14(5) verification by two people
    Answer

    Article 50(1): tell people they are interacting with an AI system, unless that is obvious. Article 50(1) covers systems 'intended to interact directly with natural persons'. A visible disclosure removes the question of what is 'obvious'.

    A tool re-reads a recruiter's finished shortlist and flags deviations from the recruiter's earlier decisions. It does not change the shortlist. Is it high-risk?

    • Yes, always: it sits in point 4(a)
    • No, never: it only reads data
    • Employment area, but Article 6(3)(c) may lift the status; the assessment is documented; ask counsel
    • Only Article 50 applies
    Answer

    Employment area, but Article 6(3)(c) may lift the status; the assessment is documented; ask counsel. It depends. Point 4(a) may apply and 6(3)(c) speaks of detecting deviations without replacing the human assessment, but a system that performs profiling is always high-risk. Those facts need a documented assessment.

    Check yourself: a tool only flags applications that are missing a required form. Which condition of Article 6(3) might let it fall outside high-risk, and what must the provider do?

    Condition (a), "intended to perform a narrow procedural task". The provider must document its assessment before the system is placed on the market or put into service (Article 6(4)).

    Where the text stands: what official EU sources said on 2026-10-22

    Source (fetched 2026-10-22) What it says, quoted Date on the source
    Regulation (EU) 2024/1689, Article 113, as published "It shall apply from 2 August 2026." Article 6(1): "shall apply from 2 August 2027." Official Journal, 12.7.2024
    Regulation (EU) 2026/1744 "(Digital Omnibus on AI)", amending Article 113 "Chapter III, Sections 1, 2, and 3, with the exception of Article 6(5), shall apply from: (i) 2 December 2027 as regards AI systems classified as high-risk pursuant to Article 6(2) and Annex III" Official Journal, 24.7.2026
    Same regulation, new Article 111(4) Synthetic-content generators placed on the market before 2 August 2026 "shall take the necessary steps in order to comply with Article 50(2) by 2 December 2026." Official Journal, 24.7.2026
    European Commission, AI Act page "The AI Act entered into force on 1 August 2024 and became applicable on 2 August 2026, with some exceptions"; Annex III rules "have been extended to 2 December 2027" "Last update 19 October 2026"
    AI Act Service Desk, timeline Under 02 Aug 2026: "Transparency rules (Article 50) start to apply." Under 02 Dec 2027: "Rules for high-risk AI systems in Annex III apply" no date shown

    How to read this: the right-hand column is the date on the source; an amending regulation adopted in July 2026 moved the 2024 dates for the high-risk rules.

    • Articles 6, 12 and 14 sit in Chapter III, Sections 1 and 2, so for Annex III systems they follow the 2 December 2027 date. Article 50 is in Chapter IV and follows the general date.
    • In the Official Journal text of 2026/1744, Article 1 inserts paragraphs 1a to 1c into Article 6 (safety components) and replaces Article 50(7). It amends none of Article 3(12), Article 6(2) to 6(4), Article 12, Article 14, Article 50(1) to 50(6) or Annex III, and the Publications Office consolidated text of 27.7.2026 (no legal effect) shows them unchanged, so the quotes here are the 2024 wording.
    • Check the consolidated text on EUR-Lex and the Commission pages before relying on any date.
    Check yourself: which quoted date applies to the Annex III high-risk requirements on the 2026-10-22 sources, and which document set it?

    2 December 2027, in Article 113 as amended by Regulation (EU) 2026/1744. The original Article 113 gave 2 August 2026 as the general date and only moved Article 6(1) later.

    Transparency: the Article 50 disclosure

    Paragraph Text, quoted Design consequence (this course's reading)
    50(1) "Providers shall ensure that AI systems intended to interact directly with natural persons are designed and developed in such a way that the natural persons concerned are informed that they are interacting with an AI system, unless this is obvious from the point of view of a natural person who is reasonably well-informed, observant and circumspect, taking into account the circumstances and the context of use." Build the disclosure into the product
    50(5) "The information ... shall be provided to the natural persons concerned in a clear and distinguishable manner at the latest at the time of the first interaction or exposure." Show it at the first interaction; log the version shown

    How to read this: the middle column is the Regulation; the right column is a design reading, not a legal conclusion.

    The Commission's AI Act page lists "Guidelines on transparency obligations for providers and deployers of certain AI systems"; read them for borderline cases.

    Mapping the frameworks to what you produce

    Practice NIST AI RMF 1.0 (NIST AI 100-1), quoted EU AI Act, quoted on this page
    Inventory GOVERN 1.6: "Mechanisms are in place to inventory AI systems and are resourced according to organizational risk priorities." Article 6(4): the provider "shall document its assessment"
    Oversight design MAP 3.5: "Processes for human oversight are defined, assessed, and documented in accordance with organizational policies from the GOVERN function." Article 14
    Incidents MANAGE 4.3: "Processes for tracking, responding to, and recovering from incidents and errors are followed and documented." Article 12(2): post-market monitoring named as a purpose of logging

    How to read this: one registry and one log can answer a NIST outcome and an Article; they are different documents with different status.

    NIST says its framework "describes four specific functions" (GOVERN, MAP, MEASURE and MANAGE) and that "Governance is designed to be a cross-cutting function".

    ISO/IEC 42001 is not mapped here: ISO's own page returned HTTP 403 on 2026-10-22, so this page makes no claim about its contents.

    Change control and incident handling

    Figure 5. The life of an AI system in the registry

    stateDiagram-v2 accTitle: AI system lifecycle in the registry accDescr: A system is proposed, classified and approved before going live. A change to the model, prompt, data or intended purpose sends it back to classification. An incident suspends it until a fix is verified. A system can be retired from live or suspended. [*] --> Proposed Proposed --> Classified: risk record written Classified --> Approved: review passes Approved --> Live: release Live --> Classified: model, prompt, data or purpose changes Live --> Suspended: incident Suspended --> Live: fix verified Suspended --> Retired: not worth fixing Live --> Retired: decommissioned Retired --> [*]

    How to read this: every change to what the card describes returns to Classified, and Suspended is a state a system can be put into without a deployment.

    Figure 6. An incident from alert to registry entry

    sequenceDiagram accTitle: Incident handling for an AI system accDescr: The eval service alerts the on-call owner that sampled agreement fell. The owner uses the policy engine to send every case to a person. The orchestrator records the switch in the audit store. The owner opens an incident in the registry and lists affected decisions from the audit store. A verified fix returns the system through change control. participant EV as Eval service participant ON as On-call owner participant PE as Policy engine participant OR as Orchestrator participant AS as Audit store participant RG as Registry EV->>ON: Alert: sampled agreement fell below the gate ON->>PE: Switch off auto-approve (kill switch) PE->>OR: Route every case to a person OR->>AS: Append: switch event, who, when ON->>RG: Open incident, set status Suspended RG->>AS: Query decisions in the affected window AS-->>RG: Decision ids and versions Note over ON,RG: The incident record lists affected decisions, not only the cause. alt Fix verified ON->>RG: Close incident through change control else Not fixable ON->>RG: Retire the system end

    How to read this: the kill switch acts first, and the audit store turns "something went wrong" into a list of affected decisions.

    Check yourself: a team swaps the model and tightens the prompt of a live system. Which state in Figure 5 does it return to, and what must happen before Live again?

    Classified. The risk record and the card describe a version, so the change is reviewed and approved again before release.

    Case files

    Under the hood: Audit logging that survives an audit

    Intuition

    An audit log is a flight recorder for decisions: written as the decision happens, readable by someone who was not there, and impossible for the people being audited to edit quietly.

    Mechanism

    Article 12(1) and 12(2). High-risk AI systems shall technically allow for the automatic recording of events (logs) over the lifetime of the system. In order to ensure a level of traceability of the functioning of a high-risk AI system that is appropriate to the intended purpose of the system, logging capabilities shall enable the recording of events relevant for: ... (b) facilitating the post-market monitoring referred to in Article 72; ...

    The field list in Article 12(3) opens "For high-risk AI systems referred to in point 1 (a), of Annex III", which is remote biometric identification. For other uses, such as employment or credit, the text above gives purposes and leaves the field list to you and counsel.

    Figure 7. One decision writes to the audit store before it is released

    sequenceDiagram accTitle: A decision writing audit entries before release accDescr: The orchestrator appends an entry when the request arrives, collects evidence from the retriever and the model gateway, asks the policy engine and the approval service, appends the evidence and the human action, and releases the decision only after the audit store acknowledges. If the audit store is down, no decision is released. participant CA as Client app participant OR as Orchestrator participant RT as Retriever participant MG as Model gateway participant PE as Policy engine participant AP as Approval service participant AS as Audit store CA->>OR: Request with request id OR->>AS: Append: received, system and version, purpose OR->>RT: Retrieve evidence RT-->>OR: Chunk ids with document versions OR->>MG: Prompt version and model id MG-->>OR: Output and usage OR->>PE: Check output PE-->>OR: Escalate to a person OR->>AP: Case with evidence AP-->>OR: Reviewer id, action, reason code OR->>AS: Append: evidence ids, output hash, policy result, human action alt Audit store acknowledges AS-->>OR: Entry hash OR-->>CA: Decision released else Audit store unavailable OR-->>CA: No decision released (fail closed) end

    How to read this: the decision is released only after the final append is acknowledged, so a decision without a record cannot exist.

    1. Append a first entry at request time, with the system and version from the registry.
    2. Store ids and hashes of evidence (chunk ids, prompt hash, output hash), not raw personal data.
    3. Append the policy result and the human action with the reviewer's id and a reason code.
    4. Release the decision after the acknowledgement; fail closed if there is none.
    5. Copy the latest chain hash to storage the application team cannot change.

    Immutability is a storage property. One documented example: S3 Object Lock "uses a write-once-read-many (WORM) model to store objects", and in compliance mode "a protected object version can't be overwritten or deleted by any user, including the root user in your AWS account" (AWS S3 User Guide, as of 2026-10-22).

    Retention pulls two ways. Article 19(1) sets a floor for provider-held logs:

    Article 19(1) (extract). ... the logs shall be kept for a period appropriate to the intended purpose of the high-risk AI system, of at least six months, unless provided otherwise in the applicable Union or national law, in particular in Union law on the protection of personal data.

    The GDPR points the other way: personal data shall be "limited to what is necessary" and "kept in a form which permits identification of data subjects for no longer than is necessary" (Regulation (EU) 2016/679, Article 5(1)(c) and (e)). Ids and hashes in the audit store, with content in a separate store that has its own retention, serve both. Counsel sets the periods.

    What this means for architects

    What changes What you do
    A record is evidence, so the write path is part of the decision Append before release and fail closed when the append fails; test that path
    Anyone who can edit the log can rewrite history Use write-once storage and a hash chain, with credentials held by a team other than the one running the application
    Retention has a floor and a ceiling Log ids and hashes in the audit store, keep content in a deletable store, and let counsel set both periods
    Audit logs differ from traces Traces serve debugging and are sampled and short-lived (page 9); the audit log is per decision and complete; keep personal data out of both unless needed (page 12)

    Under the hood: Data lineage

    Intuition

    Lineage answers "where did this answer come from?" by chaining ids from the answer back to the exact version of the source. The W3C PROV overview (a Working Group Note, 30 April 2013) defines the idea: "Provenance is information about entities, activities, and people involved in producing a piece of data or thing".

    Mechanism

    Figure 8. Lineage from a source document version to a human decision

    flowchart LR accTitle: Data lineage from source to decision accDescr: A versioned source document is ingested by a run with an id. The run produces chunks whose ids come from the document version and text. Chunks go into an index entry with an index version. A retrieval event records the chunk ids it returned. The answer cites those chunks and a person records a decision on it. SRC["Source document, version 4"] --> ING["Ingestion run (id, time)"] ING --> CH["Chunk (id from document, version, text)"] CH --> IDX["Index entry (embedding model, index version)"] IDX --> RET["Retrieval event (query, filters, chunk ids)"] RET --> ANS["Answer with citations"] ANS --> DEC["Human decision (reviewer, action)"]

    How to read this: each arrow is a record that carries an id, so a question can be answered by walking the arrows in either direction.

    1. Give every source document a version and keep superseded versions readable.
    2. Derive each chunk id from the document id, the version and the chunk text.
    3. Record the index version and embedding model with each index entry.
    4. Write the chunk ids a retrieval returned into the audit entry for that request.
    5. To assess a document change, query for every answer that cited a changed chunk.

    Worked number (illustrative): a policy has 1,200 chunks and a revision changes 18. Each answer cites 4 chunks and 12,000 answers were issued last month. If chunks were cited uniformly at random, the chance an answer cited a changed chunk is 1 - C(1182,4) / C(1200,4) = 5.87 percent, so about 705 answers need review. Real traffic is not uniform, which is why you query instead of estimating.

    What this means for architects

    What changes What you do
    Overwriting a document in place destroys lineage Version documents and keep superseded versions read-only while answers that cited them are retained
    Lineage is cheap at write time and costly later Put chunk ids and index version in the audit entry from the first release
    A new embedding model changes what an id points to Record model and index version, and re-index under a new version id (page 5)
    "Which answers used this document?" becomes a query Build and test the impact query before you need it

    Under the hood: Model and system cards

    Intuition

    A card is a nutrition label for a model or system: short, with fixed headings, saying what it is for, how it was measured and where it should not be used.

    Mechanism

    Mitchell et al. propose model cards for models; Gebru et al. propose datasheets for datasets. The left column is the paper's own headings.

    Model card section (Mitchell et al., 2019) System card equivalent
    Model Details System id, owner, model id, prompt and retrieval versions
    Intended Use, with out-of-scope uses Intended purpose copied from the registry, with out-of-scope uses
    Factors Which user groups and input types the evaluation covers
    Metrics, Evaluation Data Golden set, metrics, run id from the eval service
    Training Data Knowledge sources, ingestion and chunking settings
    Quantitative Analyses Results per segment, with intervals
    Ethical Considerations, Caveats and Recommendations Known risks, oversight design, owner, review date

    How to read this: the right column is how this course applies each heading to a whole system.

    Gebru et al.'s datasheet has seven sections: Motivation, Composition, Collection Process, Preprocessing/cleaning/labeling, Uses, Distribution and Maintenance. Answer them for each dataset you control.

    "System card" is not defined in either paper. In this course it means the same idea applied to the whole system, including prompts, retrieval, tools and oversight.

    Figure 9. A card as a release artifact

    flowchart LR accTitle: A system card as a release artifact accDescr: The registry supplies the intended purpose and the eval service supplies a run id and metrics. They are written into a versioned system card. A release gate checks that the card is complete, that its intended purpose matches the registry and that it cites an evaluation run. A pass releases the version and a fail blocks it. REG["Registry: intended purpose"] --> CARD["System card (versioned file)"] EV["Eval service: run id, metrics"] --> CARD CARD --> GATE{"Release gate: complete, purpose matches, run cited?"} GATE -- pass --> REL["Release"] GATE -- fail --> BLK["Blocked"]

    How to read this: the card is built from two sources of truth and checked by the pipeline, so it cannot drift from the system it describes.

    Mitchell et al. say the sections "may be tailored depending on the model, context, and stakeholders", and that Training Data "May not be possible to provide in practice".

    What this means for architects

    What changes What you do
    A card describes one version Keep it in the repository and update it in the same change as the prompt or model
    Intended use decides the risk tier Copy it from the registry and fail the gate on a mismatch
    Results must cover the people affected Report metrics per segment when the use involves people (page 8)
    Training data may be unknowable Record what the provider documents and write "not disclosed" where it is not

    Under the hood: Human oversight checkpoints

    Intuition

    A person can check work before the action, after it by sampling, or when the system asks. Which fits depends on how reversible the action is and how many decisions there are.

    Mechanism

    Article 14(1). High-risk AI systems shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which they are in use.

    Article 14(4), extract. ... natural persons to whom human oversight is assigned are enabled, as appropriate and proportionate: ... (b) to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias), in particular for high-risk AI systems used to provide information or recommendations for decisions to be taken by natural persons; ... (d) to decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output of the high-risk AI system; (e) to intervene in the operation of the high-risk AI system or interrupt the system through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state.

    Article 14(5) is stricter but limited by its first words: "For high-risk AI systems referred to in point 1(a) of Annex III" (remote biometric identification). There, an identification must be "separately verified and confirmed by at least two natural persons" before any action or decision is taken on it.

    Figure 10. Three checkpoint types

    sequenceDiagram accTitle: Three human oversight checkpoints accDescr: Before the action, the policy engine requires approval and a reviewer decides with evidence. By sampling, the approval service pulls a share of auto-approved outputs and records agreement. By escalation, a low confidence score or rule hit routes the case to a reviewer. All results go to the audit store. participant OR as Orchestrator participant PE as Policy engine participant AP as Approval service participant RV as Reviewer participant AS as Audit store Note over OR,AS: A. Before the action OR->>PE: Proposed action and risk tier PE-->>OR: Approval required OR->>AP: Case with evidence and confidence AP->>RV: Evidence first, then the proposal RV-->>AP: Approve, change or reject, with reason AP->>AS: Append: reviewer, time spent, action Note over OR,AS: B. By sampling, after the fact AP->>RV: Random share of auto-approved outputs RV-->>AP: Agree or disagree AP->>AS: Append: agreement rate for the period Note over OR,AS: C. By escalation PE->>AP: Low confidence or rule hit AP->>RV: Case routed to a person RV-->>AP: Decision and reason AP->>AS: Append: escalation and outcome

    How to read this: A gates a single action, B measures the automatic path after the fact, and C lets the system hand a case to a person.

    1. Gate before the action when it is hard to reverse or the tier is high.
    2. Sample the automatic path so a drifting approve rate shows up as lower agreement.
    3. Escalate on low confidence or a rule hit, sized to reviewer capacity.
    4. Record reviewer id, time spent, action and reason for every human step.
    5. Track the override rate. NIST AI 100-1 says "Data about the frequency and rationale with which humans overrule AI system output in deployed systems may be useful to collect and analyze."

    Over-reliance is documented. A systematic review in healthcare (Goddard, Roudsari and Wyatt, JAMIA 2012; abstract read on PubMed) calls automation bias "the tendency to over-rely on automation" and reports that workload and time constraint mediated it; mitigators include training and "updated confidence levels attached to DSS output". It covers clinical decision support, so treat it as a design pointer, not proof for every domain.

    How many sampled outputs show a drop in agreement? Set p1 to 0.85 and compare.

    Sampled outputs needed to detect agreement falling from 95% to 90% (interactive when JavaScript is on)
    Items to detect 95.0% falling to 90.0%149.3
    With continuity correction (n + 1/delta)169.3

    With the defaults (one-sided 5 percent, power 80 percent) the widget gives about 149 items, 169 with the continuity correction. Page 8 explains the formula.

    Check yourself: 20,000 decisions a day, 8 percent escalated, 90 seconds per review. How many hours of review does escalation alone create?

    1,600 escalations x 90 seconds = 144,000 seconds = 40 hours a day. Oversight not staffed to that number becomes a rubber stamp or a backlog.

    What this means for architects

    What changes What you do
    Oversight is a capacity problem Compute reviews per day times time per review, and set escalation thresholds from capacity
    A checkpoint that never overrides tells you little Track override rate and review time, and investigate both extremes
    The interface shapes the decision Show evidence before the recommendation, show confidence, require a reason for overrides
    The "stop" capability must work Build the kill switch into the policy engine and test it like any other control

    Decision framework

    Figure 11. Which governance path does a change take?

    flowchart TD accTitle: Governance path for a new or changed AI use accDescr: A new or changed use is first checked against the registry and added if missing. If its intended use is in an Annex III area or an Annex I product it is treated as high-risk until counsel says otherwise. Otherwise a check for talking to people leads to a disclosure, and everything else keeps a registry entry, basic logs and change control. N["New or changed AI use"] --> R{"In the registry with intended purpose?"} R -- no --> RG["Register first"] RG --> U R -- yes --> U{"Annex III area or Annex I product?"} U -- yes --> HR["Treat as high-risk until counsel says otherwise: records, oversight, monitoring"] U -- no --> TP{"Talks to people or generates content?"} TP -- yes --> DS["Add disclosure and log it"] TP -- no --> BS["Registry entry, basic logs, change control"]

    How to read this: nothing skips the registry, and the default for an Annex III use is the stricter path until a documented assessment says otherwise.

    If Then Because Revisit when
    A team wants to ship an AI feature Register it with its intended purpose before launch An unlisted system cannot be classified The purpose changes
    The intended use is in an Annex III area Treat it as high-risk and ask counsel before relying on Article 6(3) The exception needs a documented assessment and never applies to profiling Counsel's assessment is documented, or the text changes
    Decision records hold personal data Keep ids and hashes in the audit store, content elsewhere Article 19(1) and GDPR Article 5(1)(e) pull apart Retention advice changes
    A person can override the output Measure override rate, review time and sampled agreement Oversight that never overrides may be over-reliance The rates leave the expected band
    A system talks to the public Disclose at first interaction; log the version shown Article 50(1) and 50(5) A new channel is added
    The model, prompt, retrieval or purpose changes Run change control: re-evaluate, update the card, re-classify The card and risk record describe a version Every release

    Comparison of the main options

    Option Strengths Weaknesses Choose when Typical failure
    Spreadsheet registry Fast to start No checks, drifts from the code One or two low-risk systems Out of date within a quarter
    Registry as code (files in the repo, checked in CI) Reviewed with the change; checks can fail a build Needs engineering time Several systems that change often Placeholders typed in to pass the check
    Bought governance platform Workflows, reports, access control Cost, integration, the vendor's model of risk Many teams and one audit function A form-filling exercise detached from the systems
    Provider logs only No build effort You do not control retention, content or immutability Never as the sole record for a high-risk use The human step and evidence ids are missing

    Failure scenarios to design for

    Failure What the system does Why
    A system in production is not in the registry Treated as unclassified: registered by a deadline, or switched off An unlisted system has no owner, tier or log
    The audit store is unavailable Decisions that need a record are not released; others queue A decision without a record cannot be reconstructed
    A deletion request meets an immutable log The log holds ids and hashes; the content store deletes the data The two duties are met in different stores
    Reviewers approve almost everything, quickly Sampled agreement and review time raise an alert Article 14(4)(b) names the tendency to over-rely
    A source document is superseded after answers were issued The impact query lists the answers that cited it, for re-review Lineage ids make it a query
    The classification is genuinely ambiguous Use the stricter handling and escalate to counsel Article 6(3) turns on facts and a documented assessment

    Worked example

    Assumptions: 20,000 decisions a day; 6 KB per audit entry; 12 months of retention (chosen for illustration, not a legal period); 8 percent of decisions escalate to a reviewer at 90 seconds each; 2 percent of the others are sampled at 60 seconds each; a reviewer has 6 productive hours a day.

    Quantity Calculation Result
    Audit volume per day 20,000 x 6 KB 120 MB
    Audit store for 12 months 120 MB x 365 43.8 GB
    Escalated reviews per day 20,000 x 0.08 1,600
    Escalation review time 1,600 x 90 s / 3,600 40 hours
    Sampled reviews per day (20,000 - 1,600) x 0.02 368
    Sampling review time 368 x 60 s / 3,600 6.13 hours
    Reviewers needed (40 + 6.13) / 6 7.7, so 8 people
    Sample to detect agreement 95% to 90% NIST formula, one-sided 5%, power 80% about 149

    Storage is small; people are the cost. Escalating 4 percent instead of 8 halves the escalation hours to 20, a risk decision the owner records.

    Code sample: a tamper-evident audit chain

    Each entry stores the hash of the one before it, so changing an old record breaks the chain from that point. This is a sketch of the idea, not a storage design.

    import hashlib, json
    
    GENESIS = "0" * 64
    
    def digest(prev, record):
        body = json.dumps(record, sort_keys=True, separators=(",", ":"))
        return hashlib.sha256((prev + body).encode()).hexdigest()
    
    def append(log, record):
        prev = log[-1]["hash"] if log else GENESIS
        log.append({"prev": prev, "record": record, "hash": digest(prev, record)})
    
    def verify(log):
        """Index of the first entry that does not match, or None."""
        prev = GENESIS
        for i, e in enumerate(log):
            if e["prev"] != prev or e["hash"] != digest(prev, e["record"]):
                return i
            prev = e["hash"]
        return None
    
    log = []
    append(log, {"decision_id": "d-001", "event": "model_called", "prompt_version": "p-14"})
    append(log, {"decision_id": "d-001", "event": "human_decision", "reviewer": "r-207", "action": "approve"})
    append(log, {"decision_id": "d-002", "event": "model_called", "prompt_version": "p-14"})
    print(verify(log))                      # None: the chain is intact
    log[1]["record"]["action"] = "reject"   # someone edits history
    print(verify(log))                      # 1: the first broken entry
    Entry Previous hash (first 8 hex digits) Entry hash (first 8)
    0 00000000 5643452e
    1 5643452e 4cc7d04b
    2 4cc7d04b 82395585

    How to read this: each row's previous hash equals the row above's entry hash; the values come from running the sample.

    Check yourself: an attacker with write access edits entry 1 and recomputes the hash of every later entry. What does the chain itself fail to show?

    Nothing: every link is consistent again. The only evidence is that the latest hash no longer matches a copy kept elsewhere, which is why that copy goes to write-once storage under another team's control.

    Common mistakes

    • Classifying the model instead of the use. Classify each intended purpose; one model can appear in several registry rows.
    • Logging full prompts with personal data forever. Log ids and hashes; keep content in a store with its own retention.
    • A person "in the loop" who sees only the AI's answer. Show evidence first, show confidence, and record overrides with a reason.
    • Writing the card once. Version it with the system and fail the release gate when it is out of date.
    • Copying dates from a summary. Quote the official page and date-stamp it.
    • Letting the application team administer the audit log. Use separate credentials.

    Try it

    Time: 20 to 40 minutes. Pick an assistant you know, or an HR-policy chatbot.

    1. Write its registry record with the seven fields from Figure 2's table, including the intended purpose in one sentence.
    2. Classify it with Figure 3 and the quoted text. If the answer is "it depends", write down on what and whom you would ask.
    3. List the entries one decision should write to an audit store, using Figure 7.
    4. Choose one oversight checkpoint type and compute the reviewer hours it needs at your volume.
    5. Draft its system card headings.

    Done when a colleague can take your registry record and answer the three auditor questions from the real-world problem without asking you.

    Key takeaways

    • Governance is records produced while the system runs: registry, risk record, card, audit log, monitoring data.
    • The EU AI Act's classification attaches to intended use, and Article 6(3) is a documented-assessment exception, not a shortcut.
    • On the official sources read on 2026-10-22, Annex III high-risk requirements apply from 2 December 2027 and Article 50 from 2 August 2026; check again.
    • Append the audit entry before the decision is released, store ids and hashes, and keep immutability in the storage layer.
    • Lineage ids turn "which answers used this document?" into a query.
    • Oversight is a capacity and design problem: measure override rate, review time and sampled agreement. Nothing here is legal advice.

    Self-check

    1. Why is intended purpose a registry field and not a property of the model?

    Article 3(12) defines it as the use the provider intends, in context, and Annex III lists uses. The same model can land in different places.

    2. A system performs profiling of natural persons and meets an Article 6(3) condition. Is it high-risk?

    Yes: Article 6(3) ends "shall always be considered to be high-risk where the AI system performs profiling of natural persons".

    3. Which Article 12 content list applies only to remote biometric identification, and what does that mean for a credit system?

    Article 12(3), which opens "For high-risk AI systems referred to in point 1 (a), of Annex III". A credit system is still under Article 12(1) and (2), which give purposes but no field list.

    4. Why copy the latest hash of an audit chain to storage outside the application team's control?

    Someone who can write to a chain can rewrite all of it; an external copy of the latest hash is the evidence it was not.

    5. Why does a reviewer who approves 99 percent of cases in a few seconds worry an architect?

    Article 14(4)(b) names over-reliance on the output (automation bias), and the Goddard review lists workload and time constraint as mediators. High approval with short review time suggests the checkpoint is not checking.

    Further reading

    Last verified: 2026-10-22

    Next

    This page ends Part C. Part D starts with cloud platforms, and that page is not linked here yet.

    Enroll to keep reading

    Lesson 13 of 13 · AI Architect: From AI Engineer to Expert

    How AI architects turn governance into engineering: an AI system registry, EU AI Act risk tiers, audit logs, data lineage, system cards and human oversight.

    In the rest of this lesson

    • Case files
    • Under the hood: Audit logging that survives an audit
    • Under the hood: Data lineage
    • Under the hood: Model and system cards
    • Under the hood: Human oversight checkpoints
    • Decision framework

    Enrolling is free. It unlocks every lesson in AI Architect: From AI Engineer to Expert, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.

    Ask about this article

    Get answers grounded in this post. AI-generated — based on this article, and may be imperfect.

    Scaled AI Weekly

    Enjoyed this? Get more like it every Monday.

    Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.

    Get new articles by email

    Free. We email a confirmation link first, and you can unsubscribe from any email.

    Join engineers building production AI systems

    Comments