Plain-language definitions of the terms used across the AI Architect course. Each entry is short on purpose; the lesson named under it explains the idea in depth. The glossary grows with the course: terms from later parts are added as those lessons are published. Entries are in alphabetical order and each one has a stable link (for example #trust-boundary).
A
Abstention
Declining to answer because no retrieved passage is relevant enough, and saying so instead of guessing, usually with a route to a person. In the retrieval design the Retriever returns an abstain result and the Orchestrator does not call the model. See the data and retrieval lesson.
ADR
An architecture decision record: a short document that captures one architecturally significant decision and the reasoning behind it, usually with a status such as proposed, accepted or superseded. The practice was popularised by Michael Nygard's 2011 post "Documenting Architecture Decisions". See the mindset lesson.
Agent loop
A loop around a model call: call the model, and if it asked for a tool, run the tool, append the result and call the model again. A run needs exits other than the model deciding to stop (a step budget, a failure, a cancel), which the lesson draws as a state machine. See the agent loop lesson section.
AI system inventory
A maintained list of every AI system an organisation runs, with its owner, intended purpose, role, risk record, versions, evidence pointers and incidents (the lesson also calls it the registry). A system that is not listed cannot be classified, monitored or switched off. See the governance lesson.
Annex III
As used in the EU AI Act: the annex that lists uses of AI systems in eight areas, such as employment and access to essential services. Article 6(2) says AI systems referred to in Annex III "shall be considered to be high-risk", subject to the Article 6(3) conditions, and a system that performs profiling of natural persons is always high-risk. It lists uses, not models, so the tier follows the intended purpose. Not legal advice. See the governance lesson.
Answer relevance
A reference-free generation metric from the RAGAS paper: an LLM generates questions from the answer, and the score is the mean cosine similarity between those questions and the original one. It does not check facts, so a fluent, on-topic, wrong answer can score well. See the evaluation lesson.
Approval gate
A point in the flow where a risky tool call waits for a person's decision before it runs. The approval is bound to the exact call, the approver sees the exact arguments, the request expires, and a rejection or expiry goes back to the model as a tool error. See the agents, tools and MCP lesson.
Approval service
A course component name. The service that holds a risky action (for example, creating a ticket or changing access) until a person or a policy approves it. It sits between the Orchestrator and the Tool server when an action is not safe to run automatically.
Architecturally significant requirement
A requirement that has a measurable effect on the architecture and quality of a system (the definition used by the ADR community site, adr.github.io). "The button is blue" is not one; "answers must arrive within three seconds for 95 percent of questions" is. See the mindset lesson.
Audit trail
The record of one decision that lets a reviewer rebuild it later: the system and version, evidence ids, an output hash, the policy result and the human action. The course appends it before the decision is released, stores ids and hashes instead of raw personal data, and keeps it where the application team cannot edit it. It differs from a trace, which serves debugging and is sampled and short-lived. The lesson also calls it the audit log or audit store. See the governance lesson.
Automation bias
The tendency to over-rely on automation. Article 14(4)(b) of the EU AI Act asks that people assigned to oversight stay aware of "the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)". A reviewer who approves almost everything in seconds is a warning sign, so show evidence before the recommendation and measure the override rate. See the governance lesson and the hospital case.
B
Batch window
The time a provider's batch interface allows for a batch to finish before unfinished requests expire. It is provider-specific: as of 2026-10-22 the lesson's table shows Anthropic's batches expiring after 24 hours (most finish within an hour) and OpenAI's window as 24 hours. Batch only work whose deadline is longer than the window plus the time to run stragglers in real time. See the cost lesson.
BM25
A lexical ranking function that scores a document by the query words it contains, weighting rare words more, letting repeated occurrences count less and less, and discounting long documents. It needs the literal token to be present, so it finds identifiers such as clause numbers that embeddings blur, and it finds no synonyms. See the data and retrieval lesson.
Build vs buy vs configure
The three ways to get a capability: write it yourself, adopt a product or managed service, or adjust a capability you already own (for example the search in your current document system) without new software. The choice is recorded as a decision with its cost, risk and exit options. See the mindset lesson.
Bulkhead
Giving each feature (or tenant) its own pool of capacity, named for a ship's compartments, so a slow feature cannot use every slot. The lesson's example is a chat pool of 14 concurrent calls and a batch pool of 6 in front of the same provider, sized with Little's law plus headroom. Provider-side workspace or project limits work as bulkheads too. See the reliability lesson.
Byte-pair encoding
A method of building a tokenizer vocabulary (BPE): start from characters and repeatedly replace the most frequent adjacent pair with one new symbol. At use time the merges are applied in the order they were learned, so any word can be encoded. Each model family has its own merge list, so the same text gives different token counts. See the models and routing lesson.
C
C4 model
A way to describe software architecture as a set of views at different levels of detail, created by Simon Brown: a system context view, a container view, a component view and a code view, plus supporting diagrams (system landscape, dynamic and deployment). On this site the C4-style views are drawn as Mermaid flowcharts with subgraphs. See the diagramming lesson.
Cache hit rate
The share of requests that find their prefix already cached. Caching a prefix pays only above a break-even hit rate, h* = (w - 1) / (w - r) for write multiplier w and read multiplier r, which is 21.7 percent at the lesson's illustrative 1.25 and 0.1. Multipliers and lifetimes are provider-specific, so read them from your provider and track the real hit rate per route. See the cost lesson.
Canary release
A release pattern in which a new version is judged on live traffic against the current version before it is rolled out fully. It reuses the metrics and confidence intervals of offline evaluation; the release mechanics are taught in a later part of the course. See the evaluation lesson.
Chunking
Splitting source documents into the pieces that are embedded, indexed, retrieved and cited. Each chunk gets one vector and one citation, so the split decides what a vector can mean: a cut through a table row or a contract clause gives a vector that represents half of each. See the data and retrieval lesson.
Circuit breaker
A proxy that stops calling a failing dependency so callers fail fast instead of piling up behind it. In Closed it counts failures, in Open it fails every call at once while a timer runs, and in Half-Open it lets a limited number of trial calls through: success closes it and any failure reopens it. Open must return a fallback, and breakers are keyed by provider, model and region. See the reliability lesson.
Client app
A course component name. The web or mobile application a person uses; it is untrusted from the system's point of view.
Cohen's kappa
A measure of agreement between two raters that corrects for chance: (P(A) - P(E)) / (1 - P(E)). The course uses it to compare an LLM judge with human labels, because raw agreement can look high while kappa is low (87 percent raw agreement is a kappa of 0.57 in the lesson's worked example). See the evaluation lesson.
Compensating action
A business action that undoes the effect of an earlier step of a saga, such as releasing a hold or a reservation, because a process across services has no automatic rollback. Compensations run in reverse order, must be idempotent, and are retried until they succeed, with escalation to a person if they cannot. See the multi-agent and event-driven lesson.
Component view
A diagram that zooms into one container (for example the Orchestrator) and shows the parts inside it and how they depend on each other.
Confused deputy
A program that holds authority of its own and is steered by a caller's input into using it for the caller's request. In Norm Hardy's 1988 account a compiler licensed to write in its own directory, which also held the billing file, overwrote that file because a user named it as the output file. In an agent, the Orchestrator holds a credential and acts on text a model chose, so the fix is a scoped credential per request and no authority in model-written values. See the security lesson.
Consumer group
Consumers that share a group id and divide a topic's partitions between them: each partition is consumed by exactly one member of the group at a time. Parallelism is therefore capped by the number of partitions, and the group rebalances when a member joins or fails. See the multi-agent and event-driven lesson.
Consumer lag
How far a consumer group is behind the end of a partition's log, read in the lesson as the log end offset minus the committed offset. Slowness shows up as lag and not as errors, so alert on it per partition and on its trend. See the multi-agent and event-driven lesson.
Container view
A diagram of the containers of a system, meaning the applications and data stores that must be running for the system to work (in C4 the word is not about Docker), and how they communicate.
Context compaction
Replacing the older turns of a long conversation with a model-written summary when the context nears a trigger size, then continuing from the summary. It is lossy, so archive the full transcript first. Some providers offer it as a feature, and its availability is provider-specific. See the agents, tools and MCP lesson.
Context propagation
Passing the trace identifiers from one service to the next on every call so that all spans share one trace. OpenTelemetry's default propagator uses the W3C traceparent header (version, trace-id, parent-id and trace-flags). One lost header splits a trace in two, so a queue needs the header in its message headers. See the LLMOps lesson.
Context view
A diagram of the system as one box, the people who use it and the other systems it talks to.
Context window
The limit on the text a model can reference when generating a response, counting the input and the generated output together: the system prompt, every message including tool results, the tool definitions and the output. The size is model-specific, and so is the behaviour on overflow. See the models and routing lesson.
Continuous batching
Scheduling a model server one iteration at a time: a finished request leaves the batch immediately and a new one joins after the current iteration, instead of the batch staying fixed until every request in it has finished. It removes the waste of a fixed batch. See the models and routing lesson.
Cosine similarity
A measure of how close two vectors are in direction: their dot product divided by the product of their lengths, so higher is closer and length is ignored. For vectors scaled to unit length the dot product equals the cosine and L2 distance ranks results the same way. See the data and retrieval lesson.
Cost attribution
Splitting a provider bill by team, feature and release instead of by model. The gateway stamps tags derived from the authenticated identity and the route, records tokens per call, rolls them up and reconciles with the provider's usage report: the gap is untagged spend. Tags cannot attribute calls that bypass the gateway, shared costs, value or individual users. See the LLMOps lesson and the shared model bill case.
Cost per successful task
The cost of all attempts (model, retrieval, tools, retries, review and infrastructure) divided by the number of tasks that succeed. With 1,000 attempts at 3 cost units each and 800 accepted, it is 3,000 / 800 = 3.75 cost units. Failed attempts are paid for, so raising the success rate is a cost lever. See the cost lesson.
Cost unit
The placeholder unit (CU) the cost lessons use instead of prices, so every cost is a ratio or a CU and no real price appears. For real figures use the site's LLM cost estimator. See the cost lesson.
Cross-encoder
A model that reads a query and one passage together and outputs a relevance score. It is more accurate than comparing separately computed vectors, but nothing can be precomputed, so it needs one model run per pair and is used as a reranker on the top candidates of a cheaper first stage. See the data and retrieval lesson.
D
Data flow diagram
A diagram that models a system's processes, data stores, external entities and the data flows between them. Threat-modelling guidance uses it as the base on which trust boundaries are drawn. See the diagramming lesson.
Data lineage
The chain of ids that traces an answer back to the exact version of its source: source document version, ingestion run, chunk, index entry, retrieval event, answer and human decision. The lesson quotes the W3C PROV definition of provenance as "information about entities, activities, and people involved in producing a piece of data or thing". It turns "which answers used this document?" into a query. See the governance lesson.
Dead-letter queue
A place where messages that cannot be processed successfully are parked, with their key and error context, so that the line keeps moving. It needs an owner, alarms on depth and age and a replay tool, and its contents can be sensitive. With ordered flows, park by key so order is kept. See the multi-agent and event-driven lesson.
Decision log
The collection of ADRs created and maintained in a project (the term used by adr.github.io). Read in order, it shows what was decided, when and why. See the mindset lesson.
Decision record
In the governance and credit material, the entry written for one decision in the audit trail: ids and hashes of the versioned inputs (policy documents, prompt) and the human action, so the decision can be replayed, with raw text kept in the case system. The same words are also used for an ADR, which records a design choice instead (see ADR). See the credit decisions case.
Decode
The phase after prefill in which the model produces its output one token at a time, each token depending on the previous one. Each step must read the model's weights from memory again, so decode is limited by memory bandwidth and is why output tokens cost more latency than input tokens. See the models and routing lesson.
Deployment view
A diagram of where each container runs in one environment such as production (the servers, clusters, zones or managed services, and the network paths that connect them). Staging needs its own.
Design review
A structured meeting where the author of a design presents it, reviewers challenge it against the requirements, and the outcome is a recorded decision with conditions. See the mindset lesson.
Document store
A course component name. Where the source documents and their metadata live (for example object storage or a database), distinct from the Vector index built from them.
Dual LLM pattern
A design for agents that read untrusted text, proposed by Simon Willison and described by Beurer-Kellner et al. A privileged LLM has tools but sees only trusted input and plans with variable names. A quarantined LLM reads the untrusted text with no tools and returns a constrained result, and ordinary code substitutes the values into the tool call. The authors list a remaining risk: the quarantined model can still be injected, so its output can be tampered with. See the security lesson.
E
Effectively-once processing
At-least-once delivery combined with an idempotent effect, so that the effect lands once even though the message may arrive twice. It is not exactly-once delivery: for an external system such as a payments call, it is your idempotency design. See the multi-agent and event-driven lesson.
Egress allow-list
A list of the only hosts a tool or agent may contact on outbound calls, with everything else denied. It limits where data can go whatever the model decides, which closes the tool-call exfiltration path (P2), but it does not stop data leaving through rendered links or images (P1) or hidden in plain-looking text (P3). Pair it with fixed recipients and approval for new destinations. See the security lesson.
Embedding
A fixed-length list of numbers (a vector) that a model produces for a piece of text so that texts it treats as similar end up near each other. It is a learned summary of topical and semantic closeness, not a copy of the text, and it is blunt about exact strings. Vectors from different models are not comparable, so a model change means re-indexing. See the data and retrieval lesson.
ER diagram
An entity-relationship diagram: tables or entities, their attributes and the relationships between them. Used in this course for the audit log, the evaluation dataset and the prompt registry.
Error budget
The SRE book's wording is "a rate at which the SLOs can be missed", tracked daily or weekly. In practice it is the gap between 100 percent and the objective: a 99.5 percent objective leaves 0.5 percent of requests that may fail. Each failure spends part of it, and the team agrees in advance what happens when it runs out. See the mindset lesson.
Escalation rate
The share of requests passed from a cheaper step to a more expensive one. In a cascade it is the share sent on to the large model, and the cascade pays only below a break-even rate, 1 - (small + scorer) / large. The same word covers cases routed to a person, where thresholds decide the review hours. Track the rate against its break-even. See the cost lesson.
Eval service
A course component name. The service that runs a fixed set of test cases against a prompt, model or pipeline version and reports quality scores, so a release can be gated on results.
Event key
The value attached to an event (for example an order id) that decides which partition it goes to: events with the same key are written to the same partition and read in the order they were written. Choose the smallest entity whose events must stay in order. See the multi-agent and event-driven lesson.
Excessive agency
OWASP LLM06:2025: damaging actions taken in response to unexpected, ambiguous or manipulated model output, "regardless of what is causing the LLM to malfunction". OWASP names three root causes (excessive functionality, excessive permissions and excessive autonomy), and its prevention items include minimising tools and permissions and requiring user approval. See the security lesson.
F
Faithfulness
A reference-free generation metric from the RAGAS paper: the share of the statements in an answer that can be inferred from the retrieved context. A faithful answer is not necessarily true, because a stale or wrong passage can support a wrong answer. See the evaluation lesson.
Function calling
The contract between your application and a model API: you describe tools in the request, the model replies with a structured call, your code runs it and sends the result back in a new request. The model never runs anything itself. Also called tool use; field names differ by provider. See the agents, tools and MCP lesson.
G
Golden set
A versioned collection of inputs with what is needed to grade the outputs: a reference answer, facts that must appear, passages that should be retrieved, a rubric, or the end state an agent should reach. Items are tagged with slices, seeded from real failures, and a hold-out part is kept that nobody tunes against. See the evaluation lesson.
Graceful degradation
Giving a worse but honest answer when the best path cannot answer in time, instead of failing. The lesson's fallback ladder runs from a second deployment or provider, to a smaller model with the same output schema, a stored answer labelled with its age, accepting the job to deliver later and finally an honest error with a retry time. Each rung needs its own quota and a golden-set check, and the user is told the answer is degraded. See the reliability lesson.
Grouped-query attention
A variant of attention in which groups of query heads share keys and values, which shrinks the KV cache; multi-query attention is the extreme case with one key-value head. Fewer key-value heads means cheaper long context. See the models and routing lesson.
Guardrail
A small check in front of or behind the model, made of rules or a classifier model, that returns allow, block or uncertain. It is heuristic: the design-patterns paper says such filters "cannot guarantee prevention of all attacks", so keep structural layers (schema checks, the Policy engine with scoped credentials, human approval) and make the checkpoint fail closed when it gives no verdict. See the security lesson.
H
Handoff
In this course, the transfer of a task from one agent to the next with no coordinator staying in control (products use the word differently). It fits a pipeline of specialists, but context can be lost or bloated at each hop, control can loop, and nobody owns the outcome. See the multi-agent and event-driven lesson.
Hedged request
A second copy of a request that looks slow, sent to another replica after a short delay, using whichever answer arrives first and cancelling the other. Dean and Barroso describe a variant that waits until the first request has been outstanding longer than the 95th-percentile expected latency, which limits the extra load to about 5 percent. Hedge only idempotent reads, never a step with side effects, and expect extra requests and tokens. See the reliability lesson.
High-risk AI system
As used in the EU AI Act: a system classified as high-risk under Article 6, either as a product covered by Annex I law that needs third-party conformity assessment (Article 6(1)) or as a system referred to in Annex III (Article 6(2)). Article 6(3) conditions can lift the status of an Annex III system if the provider documents its assessment, but never for a system that performs profiling of natural persons. Logging (Article 12) and human oversight (Article 14) are written for high-risk systems. Not legal advice. See the governance lesson.
HNSW
Hierarchical Navigable Small World: an approximate nearest-neighbour index that links each vector to near ones and arranges the vectors in layers like a skip list. A search enters at the top, walks greedily toward the right region, drops a layer and refines. Recall is a dial set at search time, and in the lesson's worked example the vectors, not the graph, dominate memory. See the data and retrieval lesson.
Human-in-the-loop
A design in which a person can approve, reject or correct an action at a defined point. In an asynchronous flow the wait is a state in a durable record with a deadline and a defined outcome when it passes, never a thread blocked on a reply. See the agents, tools and MCP lesson and the multi-agent and event-driven lesson.
Hybrid search
Running a vector search and a lexical search for the same question and fusing the two rankings, usually with reciprocal rank fusion. The lexical side catches exact identifiers and the vector side catches paraphrases that share no words. See the data and retrieval lesson.
I
Idempotency key
A unique value a caller sends with a request so that repeating the same request (after a timeout or retry) has the effect of one request, not several. See the diagramming lesson.
Indirect prompt injection
Prompt injection where the text is written by a third party who controls a resource the system reads, such as an email, a web page, a retrieved document or a tool description, not by the primary user. NIST says it is enabled by "resource control", and the harmed party is often the user, not the attacker. It enters at the retrieved-content and tool-server entry points, and the first controls are to mark content as data, give the reading step no tools and require approval for actions. See the security lesson and the EchoLeak case.
Intended purpose
As used in the EU AI Act: Article 3(12) defines it as "the use for which an AI system is intended by the provider, including the specific context and conditions of use". Annex III lists uses, not models, so the risk tier follows the intended purpose and the registry records it per system. Not legal advice. See the governance lesson.
IVF
Inverted file index: the space is divided into cells with k-means, each vector is stored in the list of its nearest centroid, and a query looks only in the few nearest lists. It reduces how many vectors are examined, and must be trained on data. See the data and retrieval lesson.
J
Jitter
Randomness added to a retry wait so failed clients do not all return at the same moment. The AWS Builders' Library says it "adds some amount of randomness to the backoff to spread the retries around in time". Full jitter waits a random time between zero and the backoff window, and every retry loop still needs an attempt limit. See the reliability lesson.
JSON-RPC
The message format MCP is built on (version 2.0): a request has a method and an id and gets a response with either a result or an error, and a notification has no id and gets no reply. See the agents, tools and MCP lesson.
K
Kill switch
A flag the Policy engine checks on every call, so that an operator can stop one tool, one tenant or the whole agent without a deploy. See the agents, tools and MCP lesson.
KV cache
The stored keys and values of earlier positions, kept so that attention does not recompute them for every new token. It grows by a fixed amount per token (2 x layers x KV heads x head dimension x bytes per value), so it turns context length into memory and limits concurrency. See the models and routing lesson.
L
Least privilege
Giving a tool or an agent only the permissions its task needs. In the course that means read-only tools by default, scoped service credentials, a tool list that varies with the caller's authorisation, and writes behind an approval gate. See the agents, tools and MCP lesson and the IT helpdesk case.
Lexical index
A search index over words (for example a full-text index) that serves BM25 scoring. It sits beside the Vector index and is added to the course component names on the data and retrieval page. See the data and retrieval lesson.
Little's law
L = λ x W: the mean number of units in a system equals the arrival rate times the mean time each spends in it. In the lesson, 20 requests per second at 6 seconds per call keeps 120 calls in flight, which sizes workers and bulkhead pools. It is an average, so size for the peak; a hang to a 30 second timeout holds 600 calls where a 2 second fail-fast holds 40. See the reliability lesson.
LLM as judge
Using an ordinary model call to return a grade instead of an answer. The variants are pairwise comparison, single-answer grading and reference-guided grading. The judge is a measuring instrument with its own error, so version it, store its raw output and calibrate it against human labels. See the evaluation lesson.
LLM provider
A course component name. The external or self-hosted service that runs the language model.
M
Mean reciprocal rank
For each query, the reciprocal of the rank of the first relevant result (0 if there is none), averaged over the queries. Use it when the first result is the one that matters, and state the cutoff you use. See the evaluation lesson.
Memory store
A course component name, added in the agents lesson. Where facts that outlive a conversation are kept, scoped by tenant and user, with writes decided by the Policy engine and stored with a source and an expiry. See the agents, tools and MCP lesson.
Model card
A short document about a model with fixed headings, proposed by Mitchell et al. (2019): Model Details, Intended Use with out-of-scope uses, Factors, Metrics, Evaluation Data, Training Data, Quantitative Analyses, and Ethical Considerations with Caveats and Recommendations. The authors say the sections may be tailored. See also system card. See the governance lesson.
Model cascade
A policy that tries a cheaper model first and escalates to a larger one when the cheap answer is not reliable. An escalated request pays for both calls and waits for both, and the cascade is only as good as its scorer. See the models and routing lesson.
Model context protocol
MCP: a protocol between an AI application (the host) and the servers that provide tools, resources and prompts, carried as JSON-RPC messages. It is a different layer from function calling, and revisions differ in their lifecycle (as of 2026-10-21 the lesson reads revisions 2025-11-25 and 2026-07-28). See the agents, tools and MCP lesson.
Model gateway
A course component name. The layer that sits between the Orchestrator and one or more LLM providers and handles routing, authentication, limits, caching and logging.
Model router
The part of the Model gateway that decides which model serves a request. The three policies are rules on facts known before the call, a classifier that predicts which model is needed, and a cascade. See the models and routing lesson.
N
nDCG
Normalised discounted cumulative gain: a retrieval metric for graded relevance that discounts each result by its rank and is scaled so that a perfect ranking scores 1. Unlike recall at k, it rewards putting the most useful document first. See the evaluation lesson.
O
Offset
The position of a record in a partition's append-only log. A consumer's position is the offset of the next message to consume, and the committed offset is where a restarted consumer resumes. See the multi-agent and event-driven lesson.
Orchestrator
A course component name. The service that runs a request through the system: it calls the Retriever, the Model gateway and the Tool server, applies the Policy engine's decisions and decides what to do on failure.
Orchestrator-worker
A multi-agent design in which a lead agent coordinates the work and delegates to specialised workers that run in parallel, then merges their results. It suits tasks whose subtasks cannot be predicted, and it costs far more tokens than a single agent. See the multi-agent and event-driven lesson.
P
Paged attention
A way to store the KV cache in fixed-size blocks, with a table from logical to physical blocks. A block is allocated only when the last one fills, so each request wastes at most part of one block. It is the memory technique of the vLLM server. See the models and routing lesson.
Pairwise comparison
A judging mode in which the judge sees a question and two answers and picks the better one or declares a tie. It is usually more reliable for choosing between two versions, but it does not say whether either is good enough. Run both orders and count a win only if the same answer is preferred in both. See the evaluation lesson.
Pass at k
The probability that an agent gets at least one correct solution in k attempts (Anthropic's definition). Its counterpart, pass^k, is the probability that all k trials succeed, and is the one to report for an agent that acts on the world. See the evaluation lesson.
Policy engine
A course component name. The component that makes allow or deny decisions (who may see which data, which tool a request may call) from rules, so they are not buried in application code.
Position bias
A judge's tendency to favour an answer because of where it sits in the prompt. The mitigation is to run the comparison in both orders and count only a win that survives the swap. See the evaluation lesson.
Post-filtering
Searching everything, taking the top k and only then removing the chunks the caller may not see. It can leave few or no results for a tenant with a small share of the data, and the forbidden chunks have already been retrieved before they are removed. See the data and retrieval lesson.
Pre-filtering
Restricting the search to the chunks the caller may see, inside the index scan. How an engine implements filtering differs by product, so read its documentation and test recall per tenant size. See the data and retrieval lesson.
Prefill
The phase in which the model processes all the prompt tokens in parallel, fills the KV cache and produces the first output token. It is compute-bound, grows with prompt length, and its duration is the time to first token in the DistServe definition. See the models and routing lesson.
Product quantization
Compressing each vector into a short code by splitting it into sub-vectors and replacing each with the id of its nearest entry in a codebook. Distances are estimated from the codes, which saves memory at a measurable cost in recall, so a shortlist is usually re-scored with the original vectors. See the data and retrieval lesson.
Prompt caching
A provider feature that reuses the already-processed form of the start of a prompt. It does not change the output; it saves input processing time and cost, and works only when the start is identical from the first token. Lifetimes, minimum lengths and prices are provider-specific. See the models and routing lesson.
Prompt injection
Text in a model's context that steers it toward something its developer did not intend. It works because the model reads one input made of rules, user message, retrieved text and tool results, and roles are learned signals, not an enforced boundary: Greshake et al. say such applications "blur the line between data and instructions". No source cited in the lesson claims a defence that fully solves it, so limit the consequences structurally. See the security lesson.
Prompt registry
A store of prompt versions with names that point at them. MLflow's registry, the lesson's example, makes prompt versions immutable and uses an alias as a "mutable named reference" for the production version and for roll-backs, but its settings (model_config) stay mutable. So a registry entry alone is not a release record: pin the model, settings, tools and retrieval configuration in the release, and make a rollback a pointer move. See the LLMOps lesson.
Provider and deployer
As used in the EU AI Act: two roles defined in Article 3(3) and 3(4). A team that wraps a third-party model in its own assistant may be either or both, which is a legal question, so the registry records the answer and its date and counsel decides it. In the text the lesson quotes, Article 6(4) (documenting a not-high-risk assessment) and Article 50(1) (disclosure design) are addressed to providers. Not legal advice. See the governance lesson.
Public case
A case file based on a real organisation's published material, where every fact is attributed to the source. It is always labelled "Public case (sourced)".
Q
Quality bar
A testable statement of how good an answer must be, for example a minimum acceptance rate on a fixed evaluation set judged against a written rubric. See the mindset lesson.
Quantization
Storing a model's weights in fewer bits (for example 8 or 4 instead of 16) to cut memory and, because decode is limited by the bytes read per token, often time per token too. A quantized model is a different model, so it needs its own run on your golden set. See the models and routing lesson.
Queue-based load leveling
A pattern that puts a queue between a task and a service, so the service handles messages "at its own pace" (Azure Architecture Center). It absorbs bursts of non-urgent work but not an average arrival rate above capacity, so stamp a deadline on each job and alert on queue depth and age. A user waiting synchronously is a poor fit. See the reliability lesson.
R
Recall at k
The fraction of all relevant documents that appear in the top k results. It ignores the order inside the top k, and recall at the k you actually place in the prompt is a ceiling on what the model can ground its answer in. See the evaluation lesson.
Reciprocal rank fusion
A way to merge rankings from several retrievers using only ranks: each list adds 1 / (k + rank) for a document, with k = 60 in the original paper, so documents found by several lists add up. It ignores score magnitude, which makes the fused score a poor signal for abstention. See the data and retrieval lesson.
Redaction
Replacing sensitive values in prompts, answers and tool arguments with typed placeholders before a span is stored: pattern rules first, then entity detection. Do it in the application, before the Event bus, and if the redactor fails store metadata only. No redactor is complete (Presidio's own documentation says there is no guarantee), so keep retention limits and restricted reads. See the security lesson.
Regression gate
A check in CI that blocks a merge when evidence shows quality dropped. Decide the metric, the comparison (a fixed bar or the production version) and the noise rule in advance, and gate on a confidence bound and not on the point estimate. See the evaluation lesson.
Release router
A course component name, used in the canary release case. The Orchestrator asks it which prompt version to use: it reads versions from the Prompt registry, splits traffic by alias and segment, and is told by the Eval service to promote a version or point the alias back. See the canary case and the LLMOps lesson.
Retriever
A course component name. The component that finds the pieces of source material relevant to a question, usually by querying a Vector index and applying access filters.
Retry-after
A response header, often sent with an HTTP 429, that says how many seconds to wait before retrying. The provider pages the lesson cites say to wait that long when the header is present, and a spend-cap 429 has none and does not clear. The course treats it as a floor and adds jitter on top, which is a design choice. Signals are provider-specific. See the reliability lesson.
Retry budget
A cap on how many retries a service allows in total (for example as a share of its normal traffic), so that retries cannot multiply the load on a struggling dependency.
Retry storm
A failure pattern where many layers each retry a failing call, so one failure turns into a multiple of the original load. See the billing-dispute case.
Risk tolerance
An organisation's readiness to bear risk in order to achieve its objectives (NIST's definition). The NIST AI RMF does not prescribe it; it is set by the organisation for the context and written down. See the mindset lesson.
Rubric
The written description of what makes an answer acceptable, applied by human reviewers and used to build the judge's prompt. Write it before labelling, and have two people label subjective items and adjudicate disagreements. See the evaluation lesson.
S
Saga
A sequence of local transactions, each with a compensating action, used when one transaction across services is not possible. There is no automatic rollback and no isolation between steps, and it is coordinated either by choreography (events) or by orchestration (a coordinator). See the multi-agent and event-driven lesson.
Self-preference bias
A judge's tendency to score its own outputs higher than other texts, even when human annotators rate them as equal. When independence is cheap, use a judge from a different model family than the generator. See the evaluation lesson.
Semantic cache
A cache that returns a stored answer when a new query is close enough, by embedding distance, to an earlier one. Similar is not the same (cancelling an order and cancelling a subscription are close in meaning), and a hit can cross a tenant boundary or go stale. Scope it by tenant and content version and use it only for generic intents. See the models and routing lesson.
Sequence diagram
A diagram that shows how a request moves between participants over time, in order. The right view for any interaction with three or more participants. See the diagramming lesson.
Server-sent events
SSE: a one-way stream of events over HTTP with the media type text/event-stream, defined in the WHATWG HTML Standard and used by provider streaming APIs. Once the headers are sent the status is 200, so a failure after the first token must travel inside the stream. See the models and routing lesson.
Session store
A course component name, added in the agents lesson. The record of one conversation: its messages and run state such as the step count, kept outside the process so that any instance can continue the run. See the agents, tools and MCP lesson.
Shadow release
A release stage in which the gateway copies real requests to the candidate version and discards its responses, while its spans are traced. It shows latency, cost and format errors on real inputs, but no user reaction. Istio describes mirrored traffic as "fire and forget", and tools with side effects must be stubbed or read-only in the shadow path. See the LLMOps lesson.
Showback
Reporting each team's spend to that team without billing it, as opposed to chargeback. The cost attribution case calls showback a safer first step than billing teams, because chargeback brings politics. See the shared model bill case.
SLA
Service level agreement: a contract with users that includes consequences of meeting or missing the SLOs it contains. The SRE book's test is whether there is an explicit consequence; if not, it is an SLO. The book also advises keeping the internal SLO tighter than the one advertised to users. See the mindset lesson.
SLI
Service level indicator: a carefully defined measurement of service behaviour, such as the share of requests answered within a time limit.
SLO
Service level objective: a target value or range for an SLI over a window, for example "95 percent of requests under three seconds over 30 days".
Span
One timed unit of work in a trace. OpenTelemetry defines it as "a unit of work or operation"; it holds a name, the parent span id, start and end times, attributes (key-value metadata such as the release id or token counts), events, links and a status. See the LLMOps lesson.
Speculative decoding
A server-side technique in which a small draft model guesses the next few tokens and the large model checks them all in one pass, keeping those it would have produced and replacing the first wrong one. The output distribution is the same as the large model alone, and the gain depends on how often guesses are accepted. With an API it is not visible to you. See the models and routing lesson.
Spotlighting
A prompt-level technique from Hines et al. (arXiv 2403.14720) for marking untrusted content so the model can tell data from instructions. In the authors' GPT-family experiments attack success fell from over 50 percent to below 2 percent, a measured gain and not a guarantee, so it counts as a heuristic layer. See the security lesson.
State diagram
A diagram of the states something can be in (an agent loop, a job, a circuit breaker) and the events that move it between them.
Step budget
The cap on one agent run, made of three budgets in one: a maximum number of model calls, a maximum number of tokens and a wall-clock deadline. It is checked once per model reply, and when any of them is used up the run stops and returns what it found with a clear statement that it stopped. See the agents, tools and MCP lesson.
Structured output
A model feature that makes the output follow a JSON schema; Anthropic's documentation describes it as constrained decoding. The guarantee does not hold when the model refuses or the token limit cuts the output off, so check the stop reason before parsing. Valid JSON is not correct JSON. It is provider-specific. See the models and routing lesson.
Subagent
A specialised agent that a lead agent delegates a subtask to, usually working in its own clean context window and returning a condensed result or a reference. Cap how many a lead may start: Anthropic's research post reports early versions spawning 50 subagents for simple queries. See the multi-agent and event-driven lesson.
System card
The model card idea applied to a whole system, including prompts, retrieval, tools and oversight. Neither Mitchell et al. nor Gebru et al. define the term; this is the course's usage. It is a versioned file built from the registry (the intended purpose) and the Eval service (a run id and metrics), and a release gate fails when it is incomplete or does not match. See the governance lesson.
T
Tail latency
The latency of the slowest share of requests, read at a high percentile such as the 99th. Dean and Barroso note that if each server has a 99th-percentile latency of one second and a request needs 100 such servers in parallel, 63 percent of requests take more than one second. Fan-out lets the slowest call set the latency, and hedging cuts the tail only for idempotent, interference-driven slowness. See the reliability lesson.
Tenant isolation
The guarantee that one tenant's data never appears in another tenant's results and does not degrade another tenant's recall. The filter comes from the verified identity and is applied inside the Retriever or below it, large tenants get their own partition, and tests try to cross tenants. See the data and retrieval lesson and the multi-tenant retrieval case.
Text-to-SQL
Generating a SQL query from a question in natural language. The course's public case on Uber's QueryGPT splits the work into narrow agents so that only the relevant schema reaches the model's limited context. See the QueryGPT case.
Time per output token
TPOT: the average time per token after the first. Total latency is the time to first token plus TPOT times the number of tokens after the first. See the models and routing lesson.
Time to first token
TTFT: the wait until the first output token arrives. The DistServe paper defines it as the duration of the prefill phase, and a client also sees queueing and network time in it. It grows with prompt length. See the models and routing lesson.
Token
The unit of text a model reads and writes, and the unit in which a provider counts cost, enforces limits and measures speed. The same text is a different number of tokens in different models, so count with the target model's own tokenizer. See the models and routing lesson.
Token bucket
A rate limiter that fills at a steady rate up to a capacity: each request takes a token, and an empty bucket makes the request wait or fail, so a full bucket lets a burst through and the long-run rate is the refill. Anthropic's rate-limits page says its API uses the token bucket algorithm. Client side, set the refill to the limit divided by 60 and the capacity to the burst you accept. See the reliability lesson.
Tool definition
The description of a tool that is offered to the model: its name, a description and a schema for its parameters (the field names differ by provider). It is sent with the request, so it costs input tokens, and it is the contract the model's calls are checked against. See the agents, tools and MCP lesson.
Tool server
A course component name. The service that exposes actions (search a ticket system, create a ticket) to the Orchestrator. It is where model-chosen arguments meet real systems, so it needs its own checks.
Trace
The record of one request: a set of spans that share a trace id, arranged as a hierarchy, with the timed steps and the facts that explain them (release id, passage ids, token counts). A trace is evidence and also a data store, so message content stays opt-in and redacted. See the LLMOps lesson.
Trace store
A course component name. Where traces and logs of each request are kept so a bad answer can be reproduced and a release can be compared with the last one.
Transactional outbox
A pattern in which a service stores each outgoing message in an outbox table in the same local transaction that changes its state, and a separate relay publishes it. Messages are sent if and only if the transaction commits. The relay may publish a message twice, so consumers must be idempotent. See the multi-agent and event-driven lesson.
Trust boundary
A line in a diagram where the level of trust changes: where different entities are in control. Data and requests that cross it are checked, because the other side may be mistaken or hostile. See the diagramming lesson.
U
Unit economics
Cost per unit of useful work, instead of totals or cost per call. The cost lesson's unit is the cost per successful task, forecast as a low, base and high range and not one number. See the cost lesson.
V
Vector index
A course component name. A search index over embeddings (numeric representations of text) that returns the nearest items to a query. It is built from the Document store.
Verbosity bias
A judge's tendency to favour longer responses even when they are not clearer or more accurate. Control for response length and spot-check with humans, because a prompt change that lengthens answers can look better for the wrong reason. See the evaluation lesson.
W
Wilson interval
A confidence interval for a proportion such as a pass rate. It behaves better than the normal approximation at small sample sizes and near 0 or 100 percent, and a gate should use its lower bound and not the point estimate. See the evaluation lesson.
Worker
A course component name. A process that takes jobs from a queue or the Event bus and does the slow or asynchronous part of the work.
Last verified: 2026-10-22