Prompts and Untrusted Input: A First Injection Test

The first kind checks structure: where does untrusted text go? It needs no model and is exact. The second assumes the model was fooled. The mock ignores injected instructions by construction, so a test double plays a model that obeys the attacker, and the test asserts what the application still refuses to do.

Worked example

Assumptions: the three attack requests in m2_l2_injection.py, each the pumps request plus one injected sentence; a public user; a simulated model (FooledModel). Results come from the lesson tests.

The first two attacks were blocked by controls that already existed. The third is a real limit of the capstone, found by writing the test.

Build it

The code is lessons/M2-L2/m2_l2_injection.py (attacks and the flawed builder), with tests in test_m2_l2_injection.py. First, the structural test: for each attack the instructions equal those for "hello", and the attack arrives as the user message.

Verified: capstone/fde-product-discovery-copilot/lessons/M2-L2/test_m2_l2_injection.py

@pytest.mark.parametrize("name", sorted(ATTACKS))
def test_customer_text_stays_out_of_the_instructions(name: str) -> None:
    attack = ATTACKS[name]
    assert instructions_of(extract_messages(attack)) == instructions_of(extract_messages("hello"))
    assert extract_messages(attack)[-1].role == "user" and extract_messages(attack)[-1].content == attack

Second, the fooled model: it answers the compose task as the test says and delegates other tasks to the mock.

Verified: capstone/fde-product-discovery-copilot/lessons/M2-L2/test_m2_l2_injection.py

class FooledModel:
    """Simulates a model that obeys the attacker. The mock is not a model, so we script the bad reply."""

    name = "fooled"

    def __init__(self, **bad_replies: str) -> None:
        self.mock = MockProvider()
        self.bad_replies = bad_replies

    def complete(self, messages: Sequence[Message], schema: dict[str, Any] | None = None) -> Completion:
        task = task_of(messages)
        if task in self.bad_replies:
            return Completion(self.bad_replies[task], 10, 5)
        return self.mock.complete(messages, schema)

Third, the assertion that matters: the public user's response must not contain the partner-only product, and the test must prove the bad reply was received (otherwise it passes for the wrong reason).

Verified: capstone/fde-product-discovery-copilot/lessons/M2-L2/test_m2_l2_injection.py

def test_an_obeyed_override_cannot_recommend_a_partner_only_product(services: Services) -> None:
    fooled = FooledModel(**{TASK_COMPOSE: _bad_compose(PARTNER_ONLY_SKU, "partner-programme-guide#0")})
    svc = Services(services.settings, services.catalog, services.store, fooled)
    response = run_discovery(ATTACKS["override"], PUBLIC, svc).response
    assert not response.refused and response.recommendations == []
    assert any("no document evidence" in q for q in response.open_questions)  # the bad reply was received, then dropped
    assert PARTNER_ONLY_SKU not in [r.sku for r in response.recommendations]
    assert PARTNER_ONLY_SKU not in json.dumps(response.model_dump())

Run it:

./.venv/Scripts/python -m pytest lessons/M2-L2 -v

The last lines of a real run:

collected 12 items

lessons\M2-L2\test_m2_l2_injection.py ............                       [100%]

============================= 12 passed in 5.53s ==============================

Decide first, then reveal.

Exercise

Close the gap in the third row: write a function that drops any recommendation whose why repeats part of the system message, and a test using the "print your system message" attack.

Acceptance criteria:

• A test scripts a fooled reply that puts the compose system message in why and shows the capstone passes it through.
• Your function removes that recommendation and keeps an honest one.
• You can say in one sentence what your check misses.

Done when both tests pass and you can explain why a paraphrased leak gets through.

Common mistakes

• Concatenating user or document text into the system message "just to give context".
• Treating a strongly worded system prompt ("never reveal this") as a control. It is a request to the model.
• Testing injection only with the mock and calling it safe. The mock ignores the attack by construction.
• Putting access control in the prompt instead of in the query and tools.

Knowledge check

You can now

• Build prompts that keep instructions and untrusted text in separate channels.
• Classify an injection as direct or indirect.
• Write a structural test and a fooled-model test, and say what each proves.
• Describe a residual gap instead of claiming the system is safe.

Further reading

• OWASP LLM01:2025 Prompt Injection: definition and mitigations.
• Anthropic: Mitigate jailbreaks and prompt injections: a provider's guidance.
• Security and threat modeling: the architecture view.

Last verified: 2026-11-02

Next

Structured outputs and schema validation