---
title: "Put attack cases in every eval run · Agent Wranglers"
url: https://agentwranglers.se/playbook/ai-features/evals-and-quality/attack-cases-in-evals
id: AIF-77
level: Required
applies: ai-features
risk: takeover
updated: 2026-10-27
build: a2a6bdfed830
tokens: 1197
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Put attack cases in every eval run

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: AIF-77
- Level: Required
- Chapter: [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality.md)
- Rule: Give each AI feature's eval set at least 20 attack cases, marked critical. Fail CI when any run follows an injected instruction, leaks data it should not or calls a forbidden tool.
- Check: A script counts the cases tagged as attacks in each feature's eval set and fails under 20. CI fails when any attack case fails in any run. A branch that removes the user-text block from one prompt makes at least one attack case fail.
- Risk: An account or system taken over
- Applies when: The project uses AI models.
- How: Write cases for direct injection in the user's text and for instructions hidden in documents, web pages, tool results and files the feature reads. Add attempts to pull out the system prompt, other users' data or secrets. Assert in code on what must not happen, such as no call to a send tool or no planted canary string in the output. Run them at every prompt, model and tool change, with the repeats in [run each case several times](https://agentwranglers.se/playbook/ai-features/evals-and-quality/repeat-eval-runs.md). A prompt that passed last month can fail on a new model. The attack methods are in [resist direct prompt injection](https://agentwranglers.se/playbook/application-security/ai-features-and-agents/direct-prompt-injection.md), [resist indirect prompt injection](https://agentwranglers.se/playbook/application-security/ai-features-and-agents/indirect-prompt-injection.md) and [test prompt injection in every file type the model reads](https://agentwranglers.se/playbook/adversarial-testing/ai-agents-and-models/file-prompt-injection.md).

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### TypeScript example

```ts
test.each(attackCases)('attack case $id is resisted', async ({ input, document }) => {
  const tools = fakeTools();
  const answer = await assistant.answer({ userText: input, documents: [document], tools });
  expect(answer.text).not.toContain(CANARY_SECRET);
  expect(tools.calls.map(call => call.name)).not.toContain('send_email');
});
```

### Python example

```python
@pytest.mark.parametrize("case", ATTACK_CASES, ids=lambda case: case.id)
def test_attack_case_is_resisted(case):
    tools = fake_tools()
    answer = assistant.answer(user_text=case.input, documents=[case.document], tools=tools)
    assert CANARY_SECRET not in answer.text
    assert "send_email" not in [call.name for call in tools.calls]
```

## Sources

- [OWASP LLM01:2026 Prompt Injection](https://github.com/GenAI-Security-Project/GenAI-LLM-Top10/blob/main/2026/final/LLM01_PromptInjection.md)

## Related rules

- AIF-26 [Grow the eval set from production](https://agentwranglers.se/playbook/ai-features/evals-and-quality/evals-from-production.md) in AI features
- AIF-28 [Run each case several times](https://agentwranglers.se/playbook/ai-features/evals-and-quality/repeat-eval-runs.md) in AI features
- APS-82 [Resist direct prompt injection](https://agentwranglers.se/playbook/application-security/ai-features-and-agents/direct-prompt-injection.md) in Application security
- APS-83 [Resist indirect prompt injection](https://agentwranglers.se/playbook/application-security/ai-features-and-agents/indirect-prompt-injection.md) in Application security
- SEC-105 [Test prompt injection in every file type the model reads](https://agentwranglers.se/playbook/adversarial-testing/ai-agents-and-models/file-prompt-injection.md) in Adversarial testing

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/ai-features/evals-and-quality/attack-cases-in-evals">Agent Wranglers AI features playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Version the eval set and keep a held-out part](https://agentwranglers.se/playbook/ai-features/evals-and-quality/eval-set-versions.md)
- Next: [Grade with code before a grading model](https://agentwranglers.se/playbook/ai-features/evals-and-quality/code-graders-first.md)
- [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality.md)