AI features checklist

Every rule in the ai features playbook, by chapter, required rules first. Each links to its own page with how we built it and how to check it.

Deciding to use a model

• Required Try plain code before a model
• Required Define success before you build
• Required Design for wrong answers
• Recommended Choose the smallest model that passes
• Recommended Call models through one module

Prompts as code

• Required Keep prompts in files
• Required Keep user text out of the instructions
• Required Review and test every prompt change
• Required Log which prompt made each answer
• Recommended Show the model worked examples

Structured output and tools

• Required Ask for output that matches a schema
• Required Retry a bad answer and then fall back
• Required Give the model a way to say it does not know
• Required Design tools like a small public API
• Required Make tools that change things safe to repeat
• Recommended Return errors the model can act on

Retrieval

• Required Record the embedding model with every vector
• Required Measure retrieval on its own
• Required Cite a source for every answer
• Required Keep the index as fresh as its sources
• Required Copy access rights into the index and check them again
• Recommended Split documents along their structure
• Recommended Search by meaning and by words
• Recommended Say so when nothing relevant was found

Evals and quality

• Required Grade with code before a grading model
• Recommended Grow the eval set from production
• Recommended Run each case several times
• Recommended Measure cost and speed in every eval run
• Recommended Watch quality in production
• Recommended Have a person read a sample every week

Guardrails and human review

• Required Check output before anyone sees it
• Required Handle refusals and cut-off answers
• Required Let a person decide what matters
• Required Label AI output where people see it
• Recommended Check input before it reaches the model

Latency, cost and fallbacks

• Required Time out every model call
• Required Price every call and cap the spend
• Required Keep every request inside a token budget
• Required Keep working when the provider fails
• Recommended Stream long answers
• Recommended Put the stable part of the prompt first
• Optional Cache whole answers for repeated inputs
• Optional Send work nobody waits for as a batch

Logs, privacy and feedback

• Required Log every model call
• Required Keep prompt and answer text briefly and apart
• Required Train only on data people agreed to share
• Recommended Collect feedback on every answer

Agents in products

• Required End every agent run with a known status
• Required Record every step of every run
• Recommended Use a fixed workflow when the steps are known
• Recommended Let runs pause and resume
• Recommended Test agents by the end state