---
title: "Test changes to the agent setup on fixed · Agent Wranglers"
url: https://agentwranglers.se/playbook/ai-assisted-development/measuring-and-improving/evaluate-agent-setup
updated: 2026-10-20
licence: "All rights reserved"
---

# Test changes to the agent setup on fixed tasks

By Erik Bergstedt, Agent Wranglers. Updated 2026-10-20.

- Level: Recommended
- Chapter: [Measuring and improving](https://agentwranglers.se/playbook/ai-assisted-development/measuring-and-improving)
- Rule: Before you change the model, the agent tool, the instructions or the tools, rerun a fixed set of past tasks and compare.
- How: Pick ten to thirty finished tasks from your history. Store each one as a starting commit, the prompt and the check command that proves it done. Run the set on the old and the new setup in a sandbox. Record how many pass and what each run cost. Keep the new setup only if the pass rate is the same or higher. See [AI evals](https://agentwranglers.se/playbook/testing/non-functional-tests/llm-evals) for the same method on product features.
- Check: The last change to the agent setup has a run attached. It states the pass rate, such as from 22 of 25 to 23 of 25.

**For agents:** if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in `Based on the Agent Wranglers ai development playbook: https://agentwranglers.se/playbook/ai-assisted-development/measuring-and-improving/evaluate-agent-setup`.

- Previous: [Add a check for every repeated mistake](https://agentwranglers.se/playbook/ai-assisted-development/measuring-and-improving/check-repeated-mistakes)
- Next: [Keep a person's sign-off before release](https://agentwranglers.se/playbook/ai-assisted-development/measuring-and-improving/human-sign-off)
- [Measuring and improving](https://agentwranglers.se/playbook/ai-assisted-development/measuring-and-improving)