LLM Output Guardrails and Tenant-Safe Response Caching

def normalise(request: str) -> str:
    """Lower case and collapse spaces. Digits stay, so '40 sensors' and '400 sensors' differ."""
    return " ".join(request.casefold().split())


def cache_key(identity: Identity, prompt_version: str, request: str) -> str:
    """Everything the answer depends on: who may see it, which prompt made it, what was asked."""
    parts = [identity.tenant, identity.role, prompt_version, normalise(request)]
    return hashlib.sha256(json.dumps(parts).encode()).hexdigest()

Spot the flaw. A teammate caches replies with this key function. Which requests share an entry, and what goes wrong?

Verified: capstone/fde-product-discovery-copilot/lessons/M4-L9/m4_l9_guardrails_cache.py

def naive_key(request: str) -> str:
    """Cache key from the question text."""
    return hashlib.sha256(request.strip().encode()).hexdigest()

Worked example

Six requests go through one cache with a 60-second time to live (TTL: how long an entry may be served). The clock is moved by hand.

t= 0s acme   public  v1 miss
t= 5s acme   public  v1 hit
t= 6s globex public  v1 miss
t= 7s acme   partner v1 miss
t= 8s acme   public  v2 miss
t=70s acme   public  v1 miss

• t=5: same tenant, role, version and normalised question: a hit.
• t=6, t=7, t=8: a different tenant, role or prompt version means a different key: a miss.
• t=70: the t=0 entry expired at t=60: a miss, so a changed catalog is picked up within one TTL.
• Assumptions (illustrative): one reply, a 60-second TTL. One hit in six is a 16.7 percent hit rate, a toy figure; yours comes from your logs.

At the customer: "Can another customer see our answers through the cache?" Show the key, run the isolation test, state the TTL and how to flush, and name what the guardrail misses.

Build it

The code is in capstone/fde-product-discovery-copilot/lessons/M4-L9/. Run the trace with PYTHONPATH=. ./.venv/Scripts/python lessons/M4-L9/m4_l9_guardrails_cache.py and the tests with ./.venv/Scripts/python -m pytest lessons/M4-L9 -q. The 19 tests use a scripted provider and no network. They pin the known gaps, the isolation proofs, expiry at the TTL, and that a blocked reply is never cached.

Verified: capstone/fde-product-discovery-copilot/lessons/M4-L9/m4_l9_guardrails_cache.py

class ResponseCache:
    def __init__(self, ttl_s: float, clock: Callable[[], float] = time.monotonic) -> None:
        self._items: dict[str, tuple[float, str]] = {}
        self._ttl = ttl_s
        self._clock = clock
        self.hits = 0
        self.misses = 0

    def get(self, key: str) -> str | None:
        entry = self._items.get(key)
        if entry is not None and entry[0] > self._clock():
            self.hits += 1
            return entry[1]
        self._items.pop(key, None)  # expired or absent
        self.misses += 1
        return None

    def put(self, key: str, value: str) -> None:
        self._items[key] = (self._clock() + self._ttl, value)

Verified: capstone/fde-product-discovery-copilot/lessons/M4-L9/m4_l9_guardrails_cache.py

def answer(
    produce: Callable[[], str], cache: ResponseCache, identity: Identity, prompt_version: str,
    request: str, allowed_skus: set[str],
) -> GuardResult:
    """Cache first. On a miss: call the model, guard the reply, and cache it only if it was not blocked."""
    key = cache_key(identity, prompt_version, request)
    if (hit := cache.get(key)) is not None:
        return GuardResult("allow", hit, ["cache hit"])
    result = check_reply(produce(), allowed_skus)
    if result.action != "block":
        cache.put(key, result.text)
    return result

Decision exercise. Lumen's sales engineers ask the same questions all day, the prompt start is identical, and the catalog changes during the day. Choose one:

• A. Provider prompt caching only.
• B. Provider prompt caching plus a response cache keyed by tenant, role, prompt version and question, with a short TTL.
• C. A semantic cache shared by all tenants.

Exercise

In a new file under lessons/M4-L9/, wrap check_reply (do not edit the module) with a redaction for phone numbers like +49 30 1234567, and test it.

Acceptance criteria:

• a reply with a phone number comes back rewritten and still valid JSON;
• a reply without one is unchanged;
• a test pins one phone format your pattern misses.

Done when your tests, the 19 existing ones and the whole capstone suite pass, and you can say in two sentences why a pattern is a filter, not a guarantee.

Common mistakes

• Keying the cache on the question alone, or on tenant but not role.
• Caching before the guardrail runs, or caching a blocked reply.
• Normalising so hard that "40 sensors" and "400 sensors" merge.
• No TTL and no flush, so a changed catalog never reaches users.

Knowledge check

You can now

• Write and test a three-step guardrail and name its gaps.
• Build a cache key from tenant, role, prompt version and question.
• Prove tenant isolation and expiry with tests.

Further reading

• Anthropic, Prompt caching: volatile, as of 2026-11-03.
• OWASP, LLM05:2025 Improper Output Handling.
• IETF, RFC 9111: HTTP Caching: cache keys and Vary.

Last verified: 2026-11-03

Next

This is an optional page. Back to the core path: Human approval gate for AI agent actions