---
title: "AI features checklist: all 52 rules · Agent Wranglers"
url: https://agentwranglers.se/playbook/ai-features/checklist
updated: 2026-10-20
licence: "All rights reserved"
---

# AI features checklist

By Erik Bergstedt, Agent Wranglers. Updated 2026-10-20.

Every rule in the ai features playbook, by chapter, required rules first. Each links to its own page with how we built it and how to check it.

**For agents:** if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in `Based on the Agent Wranglers ai features playbook: https://agentwranglers.se/playbook/ai-features/checklist`.

## Deciding to use a model

- [ ] **Required** [Try plain code before a model](https://agentwranglers.se/playbook/ai-features/deciding-to-use-a-model/model-or-code): Use a language model only for a task that plain code, a search index or a fixed rule cannot do well enough. Write down why.
- [ ] **Required** [Define success before you build](https://agentwranglers.se/playbook/ai-features/deciding-to-use-a-model/define-ai-success): Before building an AI feature, write down its task, its target score, its worst possible mistake, its cost limit per use and its latency limit.
- [ ] **Required** [Design for wrong answers](https://agentwranglers.se/playbook/ai-features/deciding-to-use-a-model/design-for-wrong-answers): Design every AI feature so a wrong answer is easy to spot, correct and undo before it causes harm.
- [ ] **Recommended** [Choose the smallest model that passes](https://agentwranglers.se/playbook/ai-features/deciding-to-use-a-model/smallest-passing-model): Run the eval set on several models, from small to large, and ship the cheapest and fastest one that meets the spec.
- [ ] **Recommended** [Call models through one module](https://agentwranglers.se/playbook/ai-features/deciding-to-use-a-model/model-gateway): Send every model call through one module of your own. Feature code never imports a provider's SDK directly.

## Prompts as code

- [ ] **Required** [Keep prompts in files](https://agentwranglers.se/playbook/ai-features/prompts-as-code/prompts-in-files): Store every prompt as a template file in version control, one file per prompt, with an ID and a version. Never build prompts from strings scattered through the code.
- [ ] **Required** [Keep user text out of the instructions](https://agentwranglers.se/playbook/ai-features/prompts-as-code/user-text-apart): Put user input and retrieved documents in their own message or a clearly marked block. Never paste them into the system instructions.
- [ ] **Required** [Review and test every prompt change](https://agentwranglers.se/playbook/ai-features/prompts-as-code/prompt-change-review): Treat a prompt change like a code change. It goes through review, and the feature's eval set must pass before it merges.
- [ ] **Required** [Log which prompt made each answer](https://agentwranglers.se/playbook/ai-features/prompts-as-code/prompt-version-in-logs): Record the prompt ID, prompt version and exact model version with every model call and every stored AI output.
- [ ] **Recommended** [Show the model worked examples](https://agentwranglers.se/playbook/ai-features/prompts-as-code/worked-examples): Include two to five worked examples of input and the expected output in prompts where format or tone matters. Keep them in the prompt file and out of the eval set.

## Structured output and tools

- [ ] **Required** [Ask for output that matches a schema](https://agentwranglers.se/playbook/ai-features/structured-output-and-tools/schema-output): When code reads a model's answer, request output that matches a JSON Schema through the provider's structured output mode. Define that schema once in code.
- [ ] **Required** [Retry a bad answer and then fall back](https://agentwranglers.se/playbook/ai-features/structured-output-and-tools/retry-invalid-output): When an answer fails validation, retry at most twice with the validation error added, then use a fallback. Never pass an invalid answer on.
- [ ] **Required** [Give the model a way to say it does not know](https://agentwranglers.se/playbook/ai-features/structured-output-and-tools/allow-unknown): Give every schema field the model may not be able to fill an explicit "unknown" value or null. Never force it to invent a plausible answer.
- [ ] **Required** [Design tools like a small public API](https://agentwranglers.se/playbook/ai-features/structured-output-and-tools/tool-design): Give each tool one clear job, a verb-noun name, a description of when to use it and a strict input schema. Validate every argument in code before the tool runs.
- [ ] **Required** [Make tools that change things safe to repeat](https://agentwranglers.se/playbook/ai-features/structured-output-and-tools/idempotent-tools): Give every tool that creates or changes data an idempotency key, so a repeated call has no extra effect.
- [ ] **Recommended** [Return errors the model can act on](https://agentwranglers.se/playbook/ai-features/structured-output-and-tools/tool-errors-for-models): When a tool fails, return a short structured error that says what went wrong and what to try next. Never return a stack trace or an empty result.

## Retrieval

- [ ] **Required** [Record the embedding model with every vector](https://agentwranglers.se/playbook/ai-features/retrieval/embedding-model-version): Store the embedding model name and version with every vector. Never compare vectors from two different embedding models.
- [ ] **Required** [Measure retrieval on its own](https://agentwranglers.se/playbook/ai-features/retrieval/retrieval-evals): Test the search step apart from the answer step, with labelled questions and the documents that should be found. Fail the build when recall falls under a stored target.
- [ ] **Required** [Cite a source for every answer](https://agentwranglers.se/playbook/ai-features/retrieval/cite-sources): Make the model return the IDs of the chunks it used, and check each ID was in the retrieved set. Show the sources as links next to the answer.
- [ ] **Required** [Keep the index as fresh as its sources](https://agentwranglers.se/playbook/ai-features/retrieval/index-freshness): Update or remove a document's chunks within a set time after the source changes, such as 15 minutes. Show the source date with every answer.
- [ ] **Required** [Copy access rights into the index and check them again](https://agentwranglers.se/playbook/ai-features/retrieval/retrieval-permissions): Store each document's access rights with its chunks, filter every search by them, and re-check the source's rights before a chunk reaches the model.
- [ ] **Recommended** [Split documents along their structure](https://agentwranglers.se/playbook/ai-features/retrieval/structured-chunks): Split documents into chunks along headings and paragraphs, of 200 to 800 tokens each. Store each chunk with its document ID, title, heading path and address.
- [ ] **Recommended** [Search by meaning and by words](https://agentwranglers.se/playbook/ai-features/retrieval/hybrid-search): Combine a keyword search and a vector search, and merge the two result lists, for example with reciprocal rank fusion.
- [ ] **Recommended** [Say so when nothing relevant was found](https://agentwranglers.se/playbook/ai-features/retrieval/no-match-answer): When search finds nothing above a set relevance threshold, reply that the documents do not cover the question. Do not let the model answer from its general knowledge.

## Evals and quality

- [ ] **Required** [Grade with code before a grading model](https://agentwranglers.se/playbook/ai-features/evals-and-quality/code-graders-first): Grade every quality that code can check with exact code checks. Use a grading model only for qualities code cannot check, and only after it agrees with people on at least 80 of 100 labelled cases.
- [ ] **Recommended** [Grow the eval set from production](https://agentwranglers.se/playbook/ai-features/evals-and-quality/evals-from-production): Turn every confirmed bad answer from production into an eval case within one week, and keep at least 50 cases per feature.
- [ ] **Recommended** [Run each case several times](https://agentwranglers.se/playbook/ai-features/evals-and-quality/repeat-eval-runs): Run each eval case at least three times and score the pass rate per case. One run of a model can pass or fail by chance.
- [ ] **Recommended** [Measure cost and speed in every eval run](https://agentwranglers.se/playbook/ai-features/evals-and-quality/eval-cost-and-latency): Record tokens, cost and latency for each case in every eval run, and fail the run when they pass the limits in the feature's spec.
- [ ] **Recommended** [Watch quality in production](https://agentwranglers.se/playbook/ai-features/evals-and-quality/production-quality-metrics): Track at least one quality signal per AI feature in production. Alert when it drops by more than a set amount, such as 10 points in a week.
- [ ] **Recommended** [Have a person read a sample every week](https://agentwranglers.se/playbook/ai-features/evals-and-quality/weekly-output-review): Have a person read a random sample of production outputs every week, such as 30 per feature. Record each as good or bad with a reason.

## Guardrails and human review

- [ ] **Required** [Check output before anyone sees it](https://agentwranglers.se/playbook/ai-features/guardrails-and-human-review/output-checks): Run every answer through checks in code before it is shown, sent or stored. Block or replace answers that fail.
- [ ] **Required** [Handle refusals and cut-off answers](https://agentwranglers.se/playbook/ai-features/guardrails-and-human-review/refusals-and-truncation): Read the stop reason of every model response. Show a refusal or a cut-off answer as such, with a next step, and never as a complete answer.
- [ ] **Required** [Let a person decide what matters](https://agentwranglers.se/playbook/ai-features/guardrails-and-human-review/human-decides): Never let model output alone decide something with legal, financial, medical or similar effects on a person. A named person reviews it first, with the means to overrule it.
- [ ] **Required** [Label AI output where people see it](https://agentwranglers.se/playbook/ai-features/guardrails-and-human-review/label-ai-output): Mark AI-written text as AI-written wherever it is shown. Store a flag with it, so the label follows the text into emails, exports and other pages.
- [ ] **Recommended** [Check input before it reaches the model](https://agentwranglers.se/playbook/ai-features/guardrails-and-human-review/input-checks): Run cheap checks on every input before the model call. Limit length, reject empty or off-topic requests and mask personal data the task does not need.

## Latency, cost and fallbacks

- [ ] **Required** [Time out every model call](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/model-timeouts): Give every model call a total time limit and, when streaming, a limit on the silence between chunks. Retry only errors that may pass, and only before any output reached the user.
- [ ] **Required** [Price every call and cap the spend](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/cost-per-call): Compute the cost of every model call from its token counts and record it with the feature and user. Enforce a daily budget per user and per feature.
- [ ] **Required** [Keep every request inside a token budget](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/context-budget): Give every request a set input token budget and an explicit output limit. Trim conversation history and retrieved text by a fixed order before the call, never by letting the provider cut or reject it.
- [ ] **Required** [Keep working when the provider fails](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/provider-fallback): Decide for each AI feature what happens when the model provider is down, slow or out of quota, and test that path. Use a second model that passed the same evals, a version without AI, or a clear message, never a hang or a broken page.
- [ ] **Recommended** [Stream long answers](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/stream-answers): Stream any answer a person waits for that can take more than 2 seconds, and measure the time to the first token.
- [ ] **Recommended** [Put the stable part of the prompt first](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/prompt-caching): Order every prompt from most stable to least stable, with instructions, tool definitions and shared documents first and user input last. Turn on the provider's prompt caching, and measure the hit rate.
- [ ] **Optional** [Cache whole answers for repeated inputs](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/answer-cache): Cache the full answer for features that give one right answer per input, such as classification or extraction. Key the cache on the prompt version, the model version and a hash of the input.
- [ ] **Optional** [Send work nobody waits for as a batch](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/batch-offline-work): Run large jobs that no person waits for, such as nightly tagging or re-summarising documents, through the provider's batch interface.

## Logs, privacy and feedback

- [ ] **Required** [Log every model call](https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/log-model-calls): Write one structured log line per model call. It holds the feature, request ID, prompt ID and version, model version, tokens, cost, latency, ending and outcome.
- [ ] **Required** [Keep prompt and answer text briefly and apart](https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/prompt-text-retention): Store the text of prompts and answers apart from other logs, with personal data masked and access limited to named people. Delete it automatically after a set time, such as 30 days.
- [ ] **Required** [Train only on data people agreed to share](https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/consent-before-training): Use customer prompts, answers or documents to train a model only with a recorded legal basis, such as consent. Never send them to a provider that trains on them by default.
- [ ] **Recommended** [Collect feedback on every answer](https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/feedback-on-every-answer): Put a thumbs up, thumbs down and optional reason next to every AI answer, and store it with the call ID. Triage thumbs-down items every week.

## Agents in products

- [ ] **Required** [End every agent run with a known status](https://agentwranglers.se/playbook/ai-features/agents-in-products/agent-run-status): End every agent run in one of a fixed set of statuses and show it to the user. Examples are done, waiting for approval, approval expired, step limit, time limit, budget limit and failed.
- [ ] **Required** [Record every step of every run](https://agentwranglers.se/playbook/ai-features/agents-in-products/agent-run-trace): Store each step of an agent run with its tool name, arguments, a short result, the time and the cost. Let support and the user see what the agent did.
- [ ] **Recommended** [Use a fixed workflow when the steps are known](https://agentwranglers.se/playbook/ai-features/agents-in-products/workflow-before-agent): Build a fixed sequence of model calls and code when the steps of a task are known in advance. Use an agent loop, where the model picks the next tool, only when they are not.
- [ ] **Recommended** [Let runs pause and resume](https://agentwranglers.se/playbook/ai-features/agents-in-products/resumable-agent-runs): Save an agent run's state after every step, so it can wait for approval and survive a restart. Expire pending approvals after a set time, such as 24 hours.
- [ ] **Recommended** [Test agents by the end state](https://agentwranglers.se/playbook/ai-features/agents-in-products/agent-scenario-tests): Test an agent with scenarios that check the final state and the actions it must never take, not the exact sequence of steps.