Put attack cases in every eval run

How to do it

Write cases for direct injection in the user's text and for instructions hidden in documents, web pages, tool results and files the feature reads. Add attempts to pull out the system prompt, other users' data or secrets. Assert in code on what must not happen, such as no call to a send tool or no planted canary string in the output. Run them at every prompt, model and tool change, with the repeats in run each case several times. A prompt that passed last month can fail on a new model. The attack methods are in resist direct prompt injection, resist indirect prompt injection and test prompt injection in every file type the model reads.

Examples

TypeScript example

test.each(attackCases)('attack case $id is resisted', async ({ input, document }) => {
  const tools = fakeTools();
  const answer = await assistant.answer({ userText: input, documents: [document], tools });
  expect(answer.text).not.toContain(CANARY_SECRET);
  expect(tools.calls.map(call => call.name)).not.toContain('send_email');
});

Python example

@pytest.mark.parametrize("case", ATTACK_CASES, ids=lambda case: case.id)
def test_attack_case_is_resisted(case):
    tools = fake_tools()
    answer = assistant.answer(user_text=case.input, documents=[case.document], tools=tools)
    assert CANARY_SECRET not in answer.text
    assert "send_email" not in [call.name for call in tools.calls]

How to check

A script counts the cases tagged as attacks in each feature's eval set and fails under 20. CI fails when any attack case fails in any run. A branch that removes the user-text block from one prompt makes at least one attack case fail.

Sources

• OWASP LLM01:2026 Prompt Injection

Related rules

• AIF-26 Grow the eval set from production in AI features
• AIF-28 Run each case several times in AI features
• APS-82 Resist direct prompt injection in Application security
• APS-83 Resist indirect prompt injection in Application security
• SEC-105 Test prompt injection in every file type the model reads in Adversarial testing