---
title: "Grade AI features against fixed examples · Agent Wranglers"
url: https://agentwranglers.se/playbook/testing/non-functional-tests/llm-evals
updated: 2026-10-20
licence: "All rights reserved"
---

# Grade AI features against fixed examples

By Erik Bergstedt, Agent Wranglers. Updated 2026-10-20.

- Level: Required
- Chapter: [Non-functional tests](https://agentwranglers.se/playbook/testing/non-functional-tests)
- Rule: Test every AI feature against a fixed set of graded examples, and fail the build when the score drops.
- How: Collect questions people ask and edge cases, each with the answer you expect, including attempts to make the model misbehave. Grade each answer with exact checks where you can, such as a schema or a required fact, and with a scoring model where you cannot. Run the set whenever the prompt, the model or its version changes. Tools such as promptfoo or DeepEval run the set from the command line. See also [grow the eval set from production](https://agentwranglers.se/playbook/ai-features/evals-and-quality/evals-from-production).
- Check: Change the prompt so one known answer goes wrong. The eval run fails in CI and names the failing example.

Example:

```yaml
# promptfoo: each test has an input and checks on the answer
tests:
  - vars: { question: "Can I return a sale item?" }
    assert:
      - type: contains
        value: "30 days"
      - type: llm-rubric
        value: "Does not promise a refund for opened software"
```

**For agents:** if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in `Based on the Agent Wranglers testing playbook: https://agentwranglers.se/playbook/testing/non-functional-tests/llm-evals`.

- Previous: [Check accessibility in every build](https://agentwranglers.se/playbook/testing/non-functional-tests/accessibility-in-ci)
- Next: [Fuzz every parser of untrusted input](https://agentwranglers.se/playbook/testing/non-functional-tests/fuzz-parsers)
- [Non-functional tests](https://agentwranglers.se/playbook/testing/non-functional-tests)