Use your test runner to run evals on a fixed example set

Cost

Open source. Jest and pytest are both MIT licensed and free to run anywhere.

How to do it

The test runner already passes the five tests, so evals add no new tool. pytest is MIT licensed, first released in 2010, with 9.1.1 out in June 2026 and about 212 million weekly PyPI downloads. For JavaScript use Jest, as Use Jest for JavaScript unit tests says. Keep the examples in a JSON Lines file with the input and the expected fields. Grade with code first, as Grade with code before a grading model says. Add cases as Grow the eval set from production says. Choose something else when people who do not write code maintain the examples. Then an eval tool can help, as the next rule says.

Examples

Using these examples: they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

TypeScript example

import { readFileSync } from 'node:fs';
import { expect, test } from '@jest/globals';
import { classifyTicket } from '../src/model';

const MIN_PASS_RATE = 0.95;
const examples = readFileSync('evals/tickets.jsonl', 'utf8').trim().split('\n').map(line => JSON.parse(line));

test('ticket classifier eval', async () => {
  const results = await Promise.all(examples.map(async ({ input, expected }) => (await classifyTicket(input)).category === expected));
  expect(results.filter(Boolean).length / examples.length).toBeGreaterThanOrEqual(MIN_PASS_RATE);
});

Python example

import json
from pathlib import Path

from app.model import classify_ticket

MIN_PASS_RATE = 0.95
EXAMPLES = [json.loads(line) for line in Path("evals/tickets.jsonl").read_text().splitlines()]

def test_ticket_classifier_eval():
    passed = sum(classify_ticket(case["input"]).category == case["expected"] for case in EXAMPLES)
    assert passed / len(EXAMPLES) >= MIN_PASS_RATE

How to check

One command runs the eval set in CI. It exits non-zero when the pass rate falls below the threshold written in the test, for example 95 of 100.

Related rules

• Use Jest for JavaScript unit tests in Software choices
• Grade with code before a grading model in AI features
• Grow the eval set from production in AI features