AI features playbook

An AI features playbook for any team that puts a language model inside its product. It covers chat assistants, summaries, search answers, extraction, drafting and agents that act for users. It works with any provider, such as OpenAI, Anthropic, Google or an open model you host yourself.

A model is a part that guesses. It gives different answers to the same question, it changes when the provider updates it, and it costs money on every call. These rules make that part testable, affordable and safe to show to customers.

Start with Deciding to use a model, because the cheapest AI feature is the one you never build. Then read Prompts as code and Evals and quality before the first release.

Related playbooks cover the other sides. The AI-assisted development playbook covers coding agents that write your software. The AI agents and models chapter shows how to attack your own AI features, and this playbook links there for every security rule.

Chapters

1. Deciding to use a model

   Use a model only where plain code cannot do the job. Write down what success means, pick the smallest model that meets it and plan for wrong answers.

   • Required 3
   • Recommended 2

2. Prompts as code

   Keep every prompt in a versioned file and user text apart from instructions. No prompt change ships without review and a passing eval run.

   • Required 4
   • Recommended 1

3. Structured output and tools

   Ask for JSON that matches a schema, validate it, retry with the error, then fall back. Design tools like a small public API with strict inputs.

   • Required 5
   • Recommended 1

4. Retrieval

   Answer from your own documents. Split them by structure, search by meaning and by words, cite every source, keep the index fresh and respect who may read what.

   • Required 5
   • Recommended 3

5. Evals and quality

   Grow the eval set from production, grade with code before a model, repeat noisy cases and watch quality, cost and speed after release.

   • Required 1
   • Recommended 5

6. Guardrails and human review

   Check what goes into a model and what comes out. Handle refusals and cut-off answers, let people decide what matters and label AI output.

   • Required 4
   • Recommended 1

7. Latency, cost and fallbacks

   Stream long answers, time out every call, cache stable prompt parts, price every call, and keep the feature useful when the provider is down.

   • Required 4
   • Recommended 2
   • Optional 2

8. Logs, privacy and feedback

   Log every model call with its versions and cost. Keep prompt text briefly and apart, collect feedback and train only on data people agreed to.

   • Required 3
   • Recommended 1

9. Agents in products

   Use a fixed workflow when the steps are known. When an agent must choose its steps, end every run with a status, keep a full trace and survive restarts.

   • Required 2
   • Recommended 3

All 52 rules on one page