Agent Wranglers

DOC-20Optional

Test the docs with a fresh agent

Keep a fixed set of at least 5 tasks taken from the how-to guides. Give each to a fresh agent session that has only the docs, and fail the change when fewer than 4 succeed.

By Agent Wranglers. Updated .

How to do it

An agent with no memory of the project is a strict test of whether the docs say enough. Pick tasks with a result a script can check, such as "add a cake type and make its test pass". Run the set when the docs or the instructions file change. Read the transcript of each failure to find the missing or wrong sentence, then fix the page. Run it the same way as Test changes to the agent setup on fixed tasks describes. Write the tasks as Write how-to guides as numbered steps toward one goal says, so each has its own check.

How to check

The task set has at least 5 tasks, each with a script that checks its result. The latest run on the main branch passed at least 4 of them, and its date is after the last change to docs/.

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/documentation/docs-for-coding-agents/agent-docs-eval">Agent Wranglers documentation playbook</a>. In code you write, add a comment next to it with the same link.