---
title: "Grade AI features against fixed examples · Agent Wranglers"
url: https://agentwranglers.se/playbook/ai-features/evals-and-quality/llm-evals
id: AIF-86
level: Recommended
applies: ai-features
updated: 2026-10-27
build: a2a6bdfed830
tokens: 961
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Grade AI features against fixed examples

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: AIF-86
- Level: Recommended
- Chapter: [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality.md)
- Rule: Test every AI feature against a fixed set of graded examples, and fail the build when the score drops.
- Check: Change the prompt so one known answer goes wrong. The eval run fails in CI and names the failing example.
- Applies when: The project uses AI models.
- How: Collect questions people ask and edge cases, each with the answer you expect, including attempts to make the model misbehave. Grade each answer with exact checks where you can, such as a schema or a required fact, and with a scoring model where you cannot. Set a pass rate, such as 95 percent, and fail below it. Run the set whenever the prompt, the model or its version changes. Tools such as promptfoo or DeepEval run the set from the command line. See also [grow the eval set from production](https://agentwranglers.se/playbook/ai-features/evals-and-quality/evals-from-production.md).

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### YAML example

```yaml
# promptfoo: each test has an input and checks on the answer
tests:
  - vars: { question: "Can I return a sale item?" }
    assert:
      - type: contains
        value: "30 days"
      - type: llm-rubric
        value: "Does not promise a refund for opened software"
```

## Sources

- [NIST AI 100-1, AI Risk Management Framework](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf)

## Related rules

- AID-50 [Test changes to the agent setup on fixed tasks](https://agentwranglers.se/playbook/ai-assisted-development/measuring-and-improving/evaluate-agent-setup.md) in AI-assisted development
- AIF-2 [Define success before you build](https://agentwranglers.se/playbook/ai-features/deciding-to-use-a-model/define-ai-success.md) in AI features
- AIF-8 [Review and test every prompt change](https://agentwranglers.se/playbook/ai-features/prompts-as-code/prompt-change-review.md) in AI features
- AIF-59 [Pin the AI model version you depend on](https://agentwranglers.se/playbook/ai-features/prompts-as-code/pin-model-versions.md) in AI features
- AIF-26 [Grow the eval set from production](https://agentwranglers.se/playbook/ai-features/evals-and-quality/evals-from-production.md) in AI features
- AIF-27 [Grade with code before a grading model](https://agentwranglers.se/playbook/ai-features/evals-and-quality/code-graders-first.md) in AI features
- STK-91 [Use Promptfoo only when it passes your own tests](https://agentwranglers.se/playbook/stack/ai-software/eval-tool-optional.md) in Software choices
- SRH-17 [Test search results with a fixed query set](https://agentwranglers.se/playbook/search/relevance/search-quality-tests.md) in Search

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/ai-features/evals-and-quality/llm-evals">Agent Wranglers AI features playbook</a>`. In code you write, add a comment next to it with the same link.

- Next: [Grow the eval set from production](https://agentwranglers.se/playbook/ai-features/evals-and-quality/evals-from-production.md)
- [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality.md)