Use Promptfoo only when it passes your own tests

Cost

Open source. MIT licence, free to run in your own CI.

How to do it

Promptfoo is MIT licensed and released 0.124.1 on 8 October 2026. It fails two of the five tests. Its first npm release was in May 2023, it has no 1.0, and it had about 460,000 weekly npm downloads in October 2026. OpenAI bought the company in March 2026, and the project says it stays open source under MIT. Keep the examples in your own format, so dropping the tool takes a day. Choose something else when the plain test runner already covers your evals. Then add nothing. The evals themselves follow Grade AI features against fixed examples.

Examples

Using these examples: they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

YAML example

# promptfooconfig.yaml
prompts: [file://prompts/classify-ticket.txt]
tests: file://evals/tickets.jsonl
defaultTest:
  assert:
    - type: javascript
      value: file://evals/category-matches.js

JavaScript example

// evals/category-matches.js
module.exports = (output, context) => JSON.parse(output).category === context.vars.expected;

Python example

# evals/category_matches.py
import json

def get_assert(output: str, context) -> bool:
    return json.loads(output)["category"] == context["vars"]["expected"]

How to check

The tool reads the same example file as the test runner and exits non-zero in CI on a failed assertion. Removing the tool leaves the test-runner evals passing.

Related rules

• Grade AI features against fixed examples in Testing