Agent Wranglers

AID-66Recommended

Use review bots as a first pass and never as the approval

Let AI review bots comment on pull requests, but never let a bot's review satisfy a required approval. Measure each month how many of each bot's comments led to a code change.

By Agent Wranglers. Updated .

How to do it

A review bot reads the diff, not the intent behind it. It misses a change that does the wrong thing well, which Check the change does only what was asked covers. Copilot's reviews do not count toward required approvals by default, but a setting lets its approvals satisfy the rule. Keep that setting off. When one model wrote the code and another reviews it, the review still counts as a fresh review, not as a person's approval. Turn off a bot, or a rule in it, when fewer than 1 in 5 of its comments led to a change over a month. Noise trains people to skip every comment.

How to check

A test pull request approved only by the bot cannot merge, because the branch rule needs one approval from a person. The monthly note lists the share of comments that led to a change for each bot.

Sources

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/ai-assisted-development/running-agents-at-scale/review-bots-first-pass">Agent Wranglers AI-assisted development playbook</a>. In code you write, add a comment next to it with the same link.