Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Security and Threat Modeling for AI Systems. It is an engineering design, not legal advice; the retention periods are the privacy team's decision, not a rule from any regulation.
Context
- A telecom's customer-care assistant handles about 120,000 conversations a day, six turns each.
- For debugging and evaluation, the platform stores full prompts and answers in the trace store for 90 days. About 140 engineers can search it.
- A nightly job copies traces into an evaluation dataset and exports them to an external observability service.
- Customers type names, phone numbers, account numbers and addresses. Assume 40% of conversations hold at least one identifier, two on average.
Problem statement
| Question from the privacy review | Finding |
|---|---|
| How many copies of raw text exist? | Three: the trace store, the evaluation dataset and the vendor export |
| How many conversations with identifiers sit at rest? | 120,000 x 40% = 48,000 a day; over 90 days, 4,320,000 |
| Who can read them? | Every engineer with trace access |
Goals (illustrative targets): raw identifiers kept for days not months, in one place, behind approval; engineers debug on redacted traces.
Constraints
- Debugging needs enough context to see why an answer went wrong.
- Evaluation needs realistic text.
- Support leads occasionally need the raw conversation for a complaint.
- Tracing must not slow the customer's request.
Why the traditional approach fell short
Masking a card number in a log line works when the value sits in a known field. Here personal data is free text inside prompts and answers, in whatever form the customer typed it, and the model can repeat it in its reply. Scanning after storage finds problems but does not un-store them. Reviewing 120,000 conversations a day by hand is not possible.
Options considered
| Option | Verdict |
|---|---|
| A. Store no content: log only ids, versions, token counts, latency and outcome (the simplest option) | Accepted as the default for most traffic; it cannot answer "why was this answer wrong?" |
| B. Store everything, restrict access | Rejected as the only control: copies spread to evaluation and the vendor, and access control does not shorten retention |
| C. Redact before storage, keep raw text in a short-lived vault, require approval to read it | Accepted where content is needed |
Solution architecture
Figure 1. Redaction before anything is stored or exported
How to read this: raw text exists in one zone only, and everything that copies or exports reads from the redacted store.
Figure 2. One span and one break-glass read
How to read this: the engineer's default view is redacted, and raw access is a separate, approved and logged path.
How AI solved it, step by step
The stepper follows one customer message through both designs.
Under the hood
The two redaction passes behave differently.
| Pass | Catches | Weakness |
|---|---|---|
| Pattern-based | Values with a fixed shape: phone numbers, account numbers of a known format, email addresses | Misses variants: digits spelled out, split across turns, or typed with unusual spacing |
| Entity-based (a model that tags names, places and similar) | Names and addresses with no fixed shape | Probabilistic: it misses some and flags some harmless text |
Neither reaches 100%, so the design must bound what slips through. At 99% recall on 96,000 identifiers a day (48,000 conversations x 2), 960 identifiers a day reach the redacted store; at 99.9% it is 96, and at 95% it is 4,800.
Where it runs matters as much as how well. Redacting inside the application, before the span is exported, keeps raw text out of the telemetry pipeline. Redact every span attribute, including tool results and model answers, not only the prompt. Details are in Redacting sensitive data from traces and logs.
What went wrong / failure modes
| Failure | What the system does | Why |
|---|---|---|
| Redaction misses an identifier | Retention (90 days) and role-based access bound the exposure, and a weekly sample of 200 redacted traces is read by a person | Redaction reduces risk, it does not remove it |
| The redactor is down | Spans are reduced to metadata and nothing raw is stored | Fail closed: a gap in traces is cheaper than a leak |
| Over-redaction hides the cause of a bug | Typed tokens keep the structure, such as the same token for the same value in a conversation | Debugging needs shape, not values |
| A tool result (an account lookup) carries personal data | It passes through the same redactor | Personal data enters through tools as well as prompts |
| A vendor SDK captures text on its own | Only the redactor's output may leave the network, enforced by egress rules | One unredacted path defeats the pipeline |
Results
These are illustrative targets, not measurements.
| Target | Before | After |
|---|---|---|
| Raw conversations with identifiers at rest | 48,000 x 90 = 4,320,000 | 48,000 x 7 = 336,000 (about 92% fewer, 1 - 7/90) |
| Copies of raw text | 3 | 1 |
| Identifiers missed at 99% recall, steady state in the redacted store | 96,000 x 90 = 8,640,000 unredacted | 960 x 90 = 86,400 |
| Canary identifiers found outside the vault | not tested | 0 |
Tradeoffs and when NOT to use this approach
- Redaction adds a component to run, tune and test, and some debugging cases will still need the raw view.
- A model-based entity pass can itself be a place where text is processed; run it inside the same boundary.
- If metadata answers your debugging questions, take option A and store no content.
- It does not decide what the law requires; ask the privacy team.
Design rules extracted
- Redact before storage and export, inside the application boundary.
- Keep raw text in one short-lived, encrypted place and make reading it an approved request.
- Build every copy (evaluation, vendors) from the redacted store only.
- Plan for missed identifiers: bound them with retention, access control and sampling.
- Test with seeded fake identifiers every night.
Assumptions
- 120,000 conversations a day; 40% hold an identifier: 120,000 x 0.40 = 48,000; two identifiers each: 96,000 a day.
- Retention: 90 days redacted, 7 days raw vault.
- Recall values (95%, 99%, 99.9%) are scenarios: 96,000 x 0.01 = 960; x 0.001 = 96; x 0.05 = 4,800.
- The 140 engineers, the weekly sample of 200 traces and all names are invented.
Last verified: 2026-10-22