---
title: "Use the coding agent that passes most of · Agent Wranglers"
url: https://agentwranglers.se/playbook/stack/ai-software/coding-agent-by-tests
updated: 2026-10-21
licence: "All rights reserved"
---

# Use the coding agent that passes most of your own task set

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers). Updated 2026-10-21.

- Level: Recommended
- Chapter: [AI software](https://agentwranglers.se/playbook/stack/ai-software)
- Rule: Pick a coding agent by running your own fixed task set on each candidate. Keep its instructions in one AGENTS.md file that any agent can read.
- Cost: Open source. Aider and Cline are Apache-2.0 agents, free to run. Agents sold by model vendors are Paid, and so is any model API an agent calls.
- How: Coding agents change every few months, and none passes the five-year test. Vendor rankings and public benchmarks do not show how an agent does on your code, so we rank none here. Run 10 to 30 of your own finished tasks on each candidate, as [Test changes to the agent setup on fixed tasks](https://agentwranglers.se/playbook/ai-assisted-development/measuring-and-improving/evaluate-agent-setup) describes. Count the passes and the cost. Include at least one open source agent, such as Aider or Cline, both Apache-2.0 licensed. Neither passes the five tests. Aider's first release was in June 2023, and its latest, 0.86.2, came out in February 2026. A paid agent is worth its cost only when it passes more of your tasks. AGENTS.md is an open format that many agents read, stewarded by the Agentic AI Foundation since December 2025. Rules written there move with you when you switch agents, as [Write one instructions file for agents](https://agentwranglers.se/playbook/ai-assisted-development/setting-up-the-project/instructions-file) says. Choose something else when an agent does not read AGENTS.md. Then give it its own file with one line that imports or points to AGENTS.md.
- Check: A decision record lists each candidate agent with its pass count and cost on the same task set. The chosen agent passed the most tasks, or the record says why not.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### TypeScript example

```ts
import { execSync } from 'node:child_process';
import { readFileSync } from 'node:fs';

const tasks = JSON.parse(readFileSync('agent-tasks.json', 'utf8'));

export function passCount(agentCommand: string): number {
  return tasks.filter(({ commit, prompt, check }) => {
    execSync(`git checkout --quiet ${commit}`);
    execSync(agentCommand, { input: prompt });
    try { execSync(check); return true; } catch { return false; }
  }).length;
}
```

### Python example

```python
import json
import subprocess
from pathlib import Path

TASKS = json.loads(Path("agent-tasks.json").read_text())

def pass_count(agent_command: list[str]) -> int:
    passed = 0
    for task in TASKS:
        subprocess.run(["git", "checkout", "--quiet", task["commit"]], check=True)
        subprocess.run(agent_command, input=task["prompt"], text=True)
        passed += subprocess.run(task["check"], shell=True).returncode == 0
    return passed
```

## Related rules

- [Test changes to the agent setup on fixed tasks](https://agentwranglers.se/playbook/ai-assisted-development/measuring-and-improving/evaluate-agent-setup) in AI development
- [Write one instructions file for agents](https://agentwranglers.se/playbook/ai-assisted-development/setting-up-the-project/instructions-file) in AI development

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/stack/ai-software/coding-agent-by-tests">Agent Wranglers software choices playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Use Promptfoo only when it passes your own tests](https://agentwranglers.se/playbook/stack/ai-software/eval-tool-optional)
- [AI software](https://agentwranglers.se/playbook/stack/ai-software)