Agent Wranglers

Prompts as code Required

Review and test every prompt change

Treat a prompt change like a code change. It goes through review, and the feature's eval set must pass before it merges.

By Erik Bergstedt, Agent Wranglers. Updated .

How to do it

Make CI run the eval set whenever a file under the prompts folder, the model setting or the output schema changes. Post the score next to the last release's score on the pull request. Block the merge when the score falls under the target in the success spec. Never edit prompts in production through an admin screen that skips this path. The method is in AI evals.

Example

# GitHub Actions: run evals when prompts, schemas or the model change
on:
  pull_request:
    paths: ["ai/prompts/**", "ai/schemas/**", "config/models.json"]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run evals -- --min-score-from ai/specs

How to check

Change one sentence in a prompt file on a branch. CI runs the eval job, and the pull request shows the score before and after. Set the target in the spec above the measured score on a branch, and the merge is blocked.

For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers ai features playbook: https://agentwranglers.se/playbook/ai-features/prompts-as-code/prompt-change-review.