---
title: "Use your test runner to run evals on a · Agent Wranglers"
url: https://agentwranglers.se/playbook/stack/ai-software/evals-in-test-runner
updated: 2026-10-21
licence: "All rights reserved"
---

# Use your test runner to run evals on a fixed example set

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers). Updated 2026-10-21.

- Level: Recommended
- Chapter: [AI software](https://agentwranglers.se/playbook/stack/ai-software)
- Rule: Run AI evals as ordinary tests in the runner the project already uses, over a fixed set of examples kept in the repository.
- Cost: Open source. Jest and pytest are both MIT licensed and free to run anywhere.
- How: The test runner already passes the five tests, so evals add no new tool. pytest is MIT licensed, first released in 2010, with 9.1.1 out in June 2026 and about 212 million weekly PyPI downloads. For JavaScript use Jest, as [Use Jest for JavaScript unit tests](https://agentwranglers.se/playbook/stack/testing-and-code-quality/jest) says. Keep the examples in a JSON Lines file with the input and the expected fields. Grade with code first, as [Grade with code before a grading model](https://agentwranglers.se/playbook/ai-features/evals-and-quality/code-graders-first) says. Add cases as [Grow the eval set from production](https://agentwranglers.se/playbook/ai-features/evals-and-quality/evals-from-production) says. Choose something else when people who do not write code maintain the examples. Then an eval tool can help, as the next rule says.
- Check: One command runs the eval set in CI. It exits non-zero when the pass rate falls below the threshold written in the test, for example 95 of 100.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### TypeScript example

```ts
import { readFileSync } from 'node:fs';
import { expect, test } from '@jest/globals';
import { classifyTicket } from '../src/model';

const MIN_PASS_RATE = 0.95;
const examples = readFileSync('evals/tickets.jsonl', 'utf8').trim().split('\n').map(line => JSON.parse(line));

test('ticket classifier eval', async () => {
  const results = await Promise.all(examples.map(async ({ input, expected }) => (await classifyTicket(input)).category === expected));
  expect(results.filter(Boolean).length / examples.length).toBeGreaterThanOrEqual(MIN_PASS_RATE);
});
```

### Python example

```python
import json
from pathlib import Path

from app.model import classify_ticket

MIN_PASS_RATE = 0.95
EXAMPLES = [json.loads(line) for line in Path("evals/tickets.jsonl").read_text().splitlines()]

def test_ticket_classifier_eval():
    passed = sum(classify_ticket(case["input"]).category == case["expected"] for case in EXAMPLES)
    assert passed / len(EXAMPLES) >= MIN_PASS_RATE
```

## Related rules

- [Use Jest for JavaScript unit tests](https://agentwranglers.se/playbook/stack/testing-and-code-quality/jest) in Software choices
- [Grade with code before a grading model](https://agentwranglers.se/playbook/ai-features/evals-and-quality/code-graders-first) in AI features
- [Grow the eval set from production](https://agentwranglers.se/playbook/ai-features/evals-and-quality/evals-from-production) in AI features

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/stack/ai-software/evals-in-test-runner">Agent Wranglers software choices playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Use the Model Context Protocol to give agents your tools](https://agentwranglers.se/playbook/stack/ai-software/mcp-for-tools)
- Next: [Use Promptfoo only when it passes your own tests](https://agentwranglers.se/playbook/stack/ai-software/eval-tool-optional)
- [AI software](https://agentwranglers.se/playbook/stack/ai-software)