Use the coding agent that passes most of your own task set

Cost

Open source. Aider and Cline are Apache-2.0 agents, free to run. Agents sold by model vendors are Paid, and so is any model API an agent calls.

How to do it

Coding agents change every few months, and none passes the five-year test. Vendor rankings and public benchmarks do not show how an agent does on your code, so we rank none here. Run 10 to 30 of your own finished tasks on each candidate, as Test changes to the agent setup on fixed tasks describes. Count the passes and the cost. Include at least one open source agent, such as Aider or Cline, both Apache-2.0 licensed. Neither passes the five tests. Aider's first release was in June 2023, and its latest, 0.86.2, came out in February 2026. A paid agent is worth its cost only when it passes more of your tasks. AGENTS.md is an open format that many agents read, stewarded by the Agentic AI Foundation since December 2025. Rules written there move with you when you switch agents, as Write one instructions file for agents says. Choose something else when an agent does not read AGENTS.md. Then give it its own file with one line that imports or points to AGENTS.md.

Examples

Using these examples: they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

TypeScript example

import { execSync } from 'node:child_process';
import { readFileSync } from 'node:fs';

const tasks = JSON.parse(readFileSync('agent-tasks.json', 'utf8'));

export function passCount(agentCommand: string): number {
  return tasks.filter(({ commit, prompt, check }) => {
    execSync(`git checkout --quiet ${commit}`);
    execSync(agentCommand, { input: prompt });
    try { execSync(check); return true; } catch { return false; }
  }).length;
}

Python example

import json
import subprocess
from pathlib import Path

TASKS = json.loads(Path("agent-tasks.json").read_text())

def pass_count(agent_command: list[str]) -> int:
    passed = 0
    for task in TASKS:
        subprocess.run(["git", "checkout", "--quiet", task["commit"]], check=True)
        subprocess.run(agent_command, input=task["prompt"], text=True)
        passed += subprocess.run(task["check"], shell=True).returncode == 0
    return passed

How to check

A decision record lists each candidate agent with its pass count and cost on the same task set. The chosen agent passed the most tasks, or the record says why not.

Related rules

• Test changes to the agent setup on fixed tasks in AI development
• Write one instructions file for agents in AI development