---
title: "Use Promptfoo only when it passes your own · Agent Wranglers"
url: https://agentwranglers.se/playbook/stack/ai-software/eval-tool-optional
updated: 2026-10-21
licence: "All rights reserved"
---

# Use Promptfoo only when it passes your own tests

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers). Updated 2026-10-21.

- Level: Optional
- Chapter: [AI software](https://agentwranglers.se/playbook/stack/ai-software)
- Rule: Add an open source eval tool, such as Promptfoo, only when it runs your example set in CI and fails the build on a drop.
- Cost: Open source. MIT licence, free to run in your own CI.
- How: Promptfoo is MIT licensed and released 0.124.1 on 8 October 2026. It fails two of the five tests. Its first npm release was in May 2023, it has no 1.0, and it had about 460,000 weekly npm downloads in October 2026. OpenAI bought the company in March 2026, and the project says it stays open source under MIT. Keep the examples in your own format, so dropping the tool takes a day. Choose something else when the plain test runner already covers your evals. Then add nothing. The evals themselves follow [Grade AI features against fixed examples](https://agentwranglers.se/playbook/testing/non-functional-tests/llm-evals).
- Check: The tool reads the same example file as the test runner and exits non-zero in CI on a failed assertion. Removing the tool leaves the test-runner evals passing.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### YAML example

```yaml
# promptfooconfig.yaml
prompts: [file://prompts/classify-ticket.txt]
tests: file://evals/tickets.jsonl
defaultTest:
  assert:
    - type: javascript
      value: file://evals/category-matches.js
```

### JavaScript example

```js
// evals/category-matches.js
module.exports = (output, context) => JSON.parse(output).category === context.vars.expected;
```

### Python example

```python
# evals/category_matches.py
import json

def get_assert(output: str, context) -> bool:
    return json.loads(output)["category"] == context["vars"]["expected"]
```

## Related rules

- [Grade AI features against fixed examples](https://agentwranglers.se/playbook/testing/non-functional-tests/llm-evals) in Testing

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/stack/ai-software/eval-tool-optional">Agent Wranglers software choices playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Use your test runner to run evals on a fixed example set](https://agentwranglers.se/playbook/stack/ai-software/evals-in-test-runner)
- Next: [Use the coding agent that passes most of your own task set](https://agentwranglers.se/playbook/stack/ai-software/coding-agent-by-tests)
- [AI software](https://agentwranglers.se/playbook/stack/ai-software)