{"playbook":"AI features","url":"https://agentwranglers.se/playbook/ai-features","updated":"2026-10-21","license":"https://creativecommons.org/licenses/by/4.0/","credit":"Required by the licence: credit Agent Wranglers with a link to the rule or playbook you used.","how":"Each rule's How text is on its own page: add .md to its url.","rules":[{"chapter":"Deciding to use a model","name":"Try plain code before a model","level":"Recommended","applies":["ai-features"],"rule":"Use a language model only for a task that plain code, a search index or a fixed rule cannot do well enough. Write down why.","check":"Each AI feature has a short decision note in the repository. It names the baseline without a model, both scores on the same eval set and the margin the model had to beat.","url":"https://agentwranglers.se/playbook/ai-features/deciding-to-use-a-model/model-or-code","related":["/playbook/compliance/ai-act/ai-register"],"sources":[]},{"chapter":"Deciding to use a model","name":"Define success before you build","level":"Recommended","applies":["ai-features"],"rule":"Before building an AI feature, write down its task, its target score, its worst possible mistake, its cost limit per use and its latency limit.","check":"The spec file exists next to the feature's code and has a target score, a worst-failure limit, a cost per request and a latency limit. CI reads the target score from it.","url":"https://agentwranglers.se/playbook/ai-features/deciding-to-use-a-model/define-ai-success","related":["/playbook/testing/non-functional-tests/llm-evals","/playbook/ai-features/prompts-as-code/prompt-change-review"],"sources":[]},{"chapter":"Deciding to use a model","name":"Choose the smallest model that passes","level":"Recommended","applies":["ai-features"],"rule":"Run the eval set on several models, from small to large, and ship the cheapest and fastest one that meets the spec.","check":"The decision note lists at least two models with score, cost and latency. The shipped model meets the spec, and no cheaper model in the list does.","url":"https://agentwranglers.se/playbook/ai-features/deciding-to-use-a-model/smallest-passing-model","related":["/playbook/operations/running-the-service/pin-model-versions","/playbook/stack/ai-software/open-weight-models"],"sources":[]},{"chapter":"Deciding to use a model","name":"Call models through one module","level":"Recommended","applies":["ai-features"],"rule":"Send every model call through one module of your own. Feature code never imports a provider's SDK directly.","check":"A search for provider SDK imports outside the gateway module finds nothing, including the `from google import genai` form. A lint rule or a test runs that search in CI.","url":"https://agentwranglers.se/playbook/ai-features/deciding-to-use-a-model/model-gateway","related":["/playbook/architecture/solid-and-dry/dependency-inversion","/playbook/ai-features/prompts-as-code/prompts-in-files","/playbook/ai-features/logs-privacy-and-feedback/log-model-calls","/playbook/stack/ai-software/official-model-sdk"],"sources":[]},{"chapter":"Deciding to use a model","name":"Design for wrong answers","level":"Recommended","applies":["ai-features"],"rule":"Design every AI feature so a wrong answer is easy to spot, correct and undo before it causes harm.","check":"For each AI feature, a browser test reaches a person from the AI screen. Where the feature drafts text, the test edits the draft before saving. Where it changes data, the test undoes the change. Every step passes.","url":"https://agentwranglers.se/playbook/ai-features/deciding-to-use-a-model/design-for-wrong-answers","related":["/playbook/ai-features/logs-privacy-and-feedback/feedback-on-every-answer","/playbook/app-ui/feedback-and-dialogs/confirm-or-undo"],"sources":[]},{"chapter":"Prompts as code","name":"Keep prompts in files","level":"Recommended","applies":["ai-features"],"rule":"Store every prompt as a template file in version control, one file per prompt, with an ID and a version. Never build prompts from strings scattered through the code.","check":"A test collects every prompt ID passed to the gateway in the code and finds a prompt file for each. The gateway accepts only a prompt ID, never prompt text. Every prompt file has an ID and a version, and a test renders each one with sample variables without error.","url":"https://agentwranglers.se/playbook/ai-features/prompts-as-code/prompts-in-files","related":["/playbook/ai-features/deciding-to-use-a-model/model-gateway","/playbook/engineering/version-control-and-review/everything-in-version-control"],"sources":[]},{"chapter":"Prompts as code","name":"Keep user text out of the instructions","level":"Required","risk":"takeover","applies":["ai-features"],"rule":"Put user input and retrieved documents in their own message or a clearly marked block. Never paste them into the system instructions.","check":"A unit test renders each prompt with user input containing `</document>` and \"ignore the instructions above\". The system message is unchanged, and the input appears only inside its own block.","url":"https://agentwranglers.se/playbook/ai-features/prompts-as-code/user-text-apart","related":["/playbook/adversarial-testing/injection/template-injection","/playbook/adversarial-testing/ai-agents-and-models/indirect-prompt-injection"],"sources":[{"label":"OWASP LLM01:2026 Prompt Injection","url":"https://github.com/GenAI-Security-Project/GenAI-LLM-Top10/blob/main/2026/final/LLM01_PromptInjection.md"}]},{"chapter":"Prompts as code","name":"Review and test every prompt change","level":"Recommended","applies":["ai-features"],"rule":"Treat a prompt change like a code change. It goes through review, and the feature's eval set must pass before it merges.","check":"Change one sentence in a prompt file on a branch. CI runs the eval job, and the pull request shows the score before and after. Set the target in the spec above the measured score on a branch, and the merge is blocked.","url":"https://agentwranglers.se/playbook/ai-features/prompts-as-code/prompt-change-review","related":["/playbook/ai-features/deciding-to-use-a-model/define-ai-success","/playbook/testing/non-functional-tests/llm-evals","/playbook/ai-features/prompts-as-code/settings-with-prompt"],"sources":[]},{"chapter":"Prompts as code","name":"Log which prompt made each answer","level":"Recommended","applies":["ai-features"],"rule":"Record the prompt ID, prompt version and exact model version with every model call and every stored AI output.","check":"Query the stored AI outputs from the last day. Every row has a prompt ID, a prompt version and a model version, and none is empty.","url":"https://agentwranglers.se/playbook/ai-features/prompts-as-code/prompt-version-in-logs","related":["/playbook/ai-features/logs-privacy-and-feedback/log-model-calls","/playbook/operations/logs/structured-logs"],"sources":[]},{"chapter":"Prompts as code","name":"Show the model worked examples","level":"Recommended","applies":["ai-features"],"rule":"Include two to five worked examples of input and the expected output in prompts where format or tone matters. Keep them in the prompt file and out of the eval set.","check":"A test confirms no example input in any prompt file also appears in that feature's eval set.","url":"https://agentwranglers.se/playbook/ai-features/prompts-as-code/worked-examples","related":["/playbook/ai-assisted-development/context-and-memory/show-good-examples"],"sources":[]},{"chapter":"Prompts as code","name":"Keep generation settings in the prompt file","level":"Recommended","applies":["ai-features"],"rule":"Store the output limit and the generation settings, such as temperature or reasoning effort, in the prompt file's front matter. Raise the prompt version when any of them changes.","check":"A test loads every prompt file and finds an output limit and every generation setting the gateway sends. A search of feature code finds no model call that sets temperature, effort or an output limit.","url":"https://agentwranglers.se/playbook/ai-features/prompts-as-code/settings-with-prompt","related":["/playbook/ai-features/prompts-as-code/prompt-change-review","/playbook/operations/running-the-service/pin-model-versions"],"sources":[]},{"chapter":"Structured output and tools","name":"Ask for output that matches a schema","level":"Recommended","applies":["ai-features"],"rule":"When code reads a model's answer, request output that matches a JSON Schema through the provider's structured output mode. Define that schema once in code.","check":"A search finds no `JSON.parse` or `json.loads` on model text outside the gateway. Every gateway call that expects data passes a schema, and a test asserts it. A test sends each schema once to every provider in use and gets no schema error.","url":"https://agentwranglers.se/playbook/ai-features/structured-output-and-tools/schema-output","related":["/playbook/adversarial-testing/ai-agents-and-models/validate-model-output","/playbook/stack/ai-software/zod-or-pydantic"],"sources":[{"label":"JSON Schema specification","url":"https://json-schema.org/specification"}]},{"chapter":"Structured output and tools","name":"Retry a bad answer and then fall back","level":"Recommended","applies":["ai-features"],"rule":"When an answer fails validation, retry at most twice with the validation error added, then use a fallback. Never pass an invalid answer on.","check":"A unit test with a fake provider returns invalid JSON three times. The gateway makes exactly three calls and returns the fallback. The failure counter shows one failed request.","url":"https://agentwranglers.se/playbook/ai-features/structured-output-and-tools/retry-invalid-output","related":["/playbook/adversarial-testing/ai-agents-and-models/validate-model-output"],"sources":[]},{"chapter":"Structured output and tools","name":"Give the model a way to say it does not know","level":"Recommended","applies":["ai-features"],"rule":"Give every schema field the model may not be able to fill an explicit \"unknown\" value or null. Never force it to invent a plausible answer.","check":"Run the eval cases that lack the needed information. Each returns `unknown` or null in the right field, and none returns an invented value.","url":"https://agentwranglers.se/playbook/ai-features/structured-output-and-tools/allow-unknown","related":["/playbook/ai-features/retrieval/no-match-answer"],"sources":[]},{"chapter":"Structured output and tools","name":"Design tools like a small public API","level":"Required","risk":"takeover","applies":["ai-features"],"rule":"Give each tool one clear job, a verb-noun name, a description of when to use it and a strict input schema. Validate every argument in code before the tool runs.","check":"A test loads every tool definition. Each has a name matching `^[a-z]+_[a-z_]+$`, a description of at least 40 characters, `additionalProperties: false` and no untyped parameter. A call with a wrong argument type is rejected before the tool body runs.","url":"https://agentwranglers.se/playbook/ai-features/structured-output-and-tools/tool-design","related":["/playbook/adversarial-testing/ai-agents-and-models/least-privilege-tools","/playbook/stack/ai-software/mcp-for-tools"],"sources":[{"label":"MCP specification on tools","url":"https://modelcontextprotocol.io/specification/2026-07-28/server/tools"},{"label":"JSON Schema specification","url":"https://json-schema.org/specification"}]},{"chapter":"Structured output and tools","name":"Return errors the model can act on","level":"Recommended","applies":["ai-features"],"rule":"When a tool fails, return a short structured error that says what went wrong and what to try next. Never return a stack trace or an empty result.","check":"A test calls each tool with a bad argument and with a missing record. Both results are short JSON objects with a code and a message, and neither contains a stack trace or file path. A test with two parallel tool calls, one failing, sends back two results.","url":"https://agentwranglers.se/playbook/ai-features/structured-output-and-tools/tool-errors-for-models","related":["/playbook/api-design/errors/problem-details"],"sources":[{"label":"MCP specification on tools","url":"https://modelcontextprotocol.io/specification/2026-07-28/server/tools"}]},{"chapter":"Structured output and tools","name":"Make tools that change things safe to repeat","level":"Required","risk":"money","applies":["ai-features"],"rule":"Give every tool that creates or changes data an idempotency key, so a repeated call has no extra effect.","check":"A test calls `create_return` twice for the same order line, with two different tool call IDs. One return exists, and both calls get the same result.","url":"https://agentwranglers.se/playbook/ai-features/structured-output-and-tools/idempotent-tools","related":["/playbook/api-design/reliability/idempotency-keys","/playbook/adversarial-testing/ai-agents-and-models/human-approval"],"sources":[{"label":"Expired IETF draft on the Idempotency-Key HTTP Header Field","url":"https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/"}]},{"chapter":"Retrieval","name":"Split documents along their structure","level":"Recommended","applies":["ai-features"],"rule":"Split documents into chunks along headings and paragraphs, of 200 to 800 tokens each. Store each chunk with its document ID, title, heading path and address.","check":"A test chunks the sample documents. Every chunk has 200 to 800 tokens or is a whole short section. Each ends at a sentence end and has a document ID, heading path and address.","url":"https://agentwranglers.se/playbook/ai-features/retrieval/structured-chunks","related":["/playbook/ai-features/retrieval/retrieval-evals"],"sources":[]},{"chapter":"Retrieval","name":"Record the embedding model with every vector","level":"Recommended","applies":["ai-features"],"rule":"Store the embedding model name and version with every vector. Never compare vectors from two different embedding models.","check":"A query counts vectors per embedding model in the live index. There is exactly one model, and it matches the model in the settings. A filtered search for a small tenant with at least 20 chunks returns 20 rows.","url":"https://agentwranglers.se/playbook/ai-features/retrieval/embedding-model-version","related":["/playbook/operations/running-the-service/pin-model-versions"],"sources":[]},{"chapter":"Retrieval","name":"Search by meaning and by words","level":"Recommended","applies":["ai-features"],"rule":"Combine a keyword search and a vector search, and merge the two result lists, for example with reciprocal rank fusion.","check":"The retrieval eval set includes questions with exact product codes. Hybrid search scores at least as well as vector search alone on the whole set.","url":"https://agentwranglers.se/playbook/ai-features/retrieval/hybrid-search","related":["/playbook/ai-features/retrieval/retrieval-evals","/playbook/stack/ai-software/pgvector-first"],"sources":[]},{"chapter":"Retrieval","name":"Measure retrieval on its own","level":"Recommended","applies":["ai-features"],"rule":"Test the search step apart from the answer step, with labelled questions and the documents that should be found. Fail the build when recall falls under a stored target.","check":"CI runs the retrieval set and fails when recall at 5 falls under the stored target, such as 0.85.","url":"https://agentwranglers.se/playbook/ai-features/retrieval/retrieval-evals","related":["/playbook/ai-features/retrieval/structured-chunks","/playbook/ai-features/retrieval/hybrid-search","/playbook/testing/non-functional-tests/llm-evals"],"sources":[]},{"chapter":"Retrieval","name":"Cite a source for every answer","level":"Recommended","applies":["ai-features"],"rule":"Make the model return the IDs of the chunks it used, and check each ID was in the retrieved set. Show the sources as links next to the answer.","check":"A unit test feeds the citation step an answer citing one retrieved and one unknown ID. Only the retrieved one is shown. An eval run shows every answer that states a fact has at least one valid citation.","url":"https://agentwranglers.se/playbook/ai-features/retrieval/cite-sources","related":["/playbook/adversarial-testing/ai-agents-and-models/grounded-answers"],"sources":[]},{"chapter":"Retrieval","name":"Say so when nothing relevant was found","level":"Recommended","applies":["ai-features"],"rule":"When search finds nothing above a set relevance threshold, reply that the documents do not cover the question. Do not let the model answer from its general knowledge.","check":"Ask a question the documents do not cover, such as one about a product you do not sell. The reply says the documents do not cover it, and the log shows a no-match event.","url":"https://agentwranglers.se/playbook/ai-features/retrieval/no-match-answer","related":["/playbook/ai-features/structured-output-and-tools/allow-unknown","/playbook/app-ui/data-views/empty-states"],"sources":[]},{"chapter":"Retrieval","name":"Keep the index as fresh as its sources","level":"Recommended","applies":["ai-features"],"rule":"Update or remove a document's chunks within a set time after the source changes, such as 15 minutes. Show the source date with every answer.","check":"Delete a test document at the source. Within the set time, a search for its unique phrase returns nothing. A nightly job compares document IDs in the source and the index and alerts on any difference.","url":"https://agentwranglers.se/playbook/ai-features/retrieval/index-freshness","related":["/playbook/data/caches-and-copies/source-of-truth","/playbook/data/caches-and-copies/search-index-sync"],"sources":[]},{"chapter":"Retrieval","name":"Copy access rights into the index and check them again","level":"Required","risk":"data","applies":["ai-features"],"rule":"Store each document's access rights with its chunks, filter every search by them, and re-check the source's rights before a chunk reaches the model.","check":"Give a test user access to a document, ask about it, then remove the access. The next question returns nothing from that document, even before the index updates.","url":"https://agentwranglers.se/playbook/ai-features/retrieval/retrieval-permissions","related":["/playbook/adversarial-testing/ai-agents-and-models/scoped-retrieval"],"sources":[{"label":"OWASP LLM09:2026 Vector and Embedding Weaknesses","url":"https://github.com/GenAI-Security-Project/GenAI-LLM-Top10/blob/main/2026/final/LLM09_VectorAndEmbeddingWeaknesses.md"}]},{"chapter":"Evals and quality","name":"Grow the eval set from production","level":"Recommended","applies":["ai-features"],"rule":"Turn every confirmed bad answer from production into an eval case within one week, and keep at least 50 cases per feature.","check":"The eval file for each feature has at least 50 cases. Every bad-answer report closed in the last month links to an eval case by ID.","url":"https://agentwranglers.se/playbook/ai-features/evals-and-quality/evals-from-production","related":["/playbook/testing/non-functional-tests/llm-evals","/playbook/ai-features/evals-and-quality/weekly-output-review","/playbook/ai-features/logs-privacy-and-feedback/feedback-on-every-answer","/playbook/stack/ai-software/evals-in-test-runner"],"sources":[]},{"chapter":"Evals and quality","name":"Grade with code before a grading model","level":"Recommended","applies":["ai-features"],"rule":"Grade every quality that code can check with exact code checks. Use a grading model only for qualities code cannot check, and only after it agrees with people on at least 80 of 100 labelled cases.","check":"Each eval case has at least one code check. The grading model's agreement with the human labels is stored and is at least 0.8.","url":"https://agentwranglers.se/playbook/ai-features/evals-and-quality/code-graders-first","related":["/playbook/testing/non-functional-tests/llm-evals","/playbook/stack/ai-software/evals-in-test-runner"],"sources":[]},{"chapter":"Evals and quality","name":"Run each case several times","level":"Recommended","applies":["ai-features"],"rule":"Run each eval case at least three times and score the pass rate per case. One run of a model can pass or fail by chance.","check":"The eval report shows each case's pass count out of the number of runs. CI fails when the overall score is under target or a case marked critical fails any run.","url":"https://agentwranglers.se/playbook/ai-features/evals-and-quality/repeat-eval-runs","related":["/playbook/testing/flaky-and-slow-tests/repeat-new-tests","/playbook/ai-features/evals-and-quality/counterfactual-evals","/playbook/ai-features/agents-in-products/agent-scenario-tests"],"sources":[]},{"chapter":"Evals and quality","name":"Measure cost and speed in every eval run","level":"Recommended","applies":["ai-features"],"rule":"Record tokens, cost and latency for each case in every eval run, and fail the run when they pass the limits in the feature's spec.","check":"The eval report shows cost and latency per case. On a branch, set the spec's cost limit to half the last measured average cost, and the eval job fails on cost.","url":"https://agentwranglers.se/playbook/ai-features/evals-and-quality/eval-cost-and-latency","related":["/playbook/operations/capacity-and-cost/cost-monitoring"],"sources":[]},{"chapter":"Evals and quality","name":"Watch quality in production","level":"Recommended","applies":["ai-features"],"rule":"Track at least one quality signal per AI feature in production. Alert when it drops by more than a set amount, such as 10 points in a week.","check":"The dashboard shows a quality signal per AI feature, split by prompt version. An alert rule exists for each and fires in a test with a simulated drop.","url":"https://agentwranglers.se/playbook/ai-features/evals-and-quality/production-quality-metrics","related":["/playbook/operations/service-levels/service-level-indicators"],"sources":[]},{"chapter":"Evals and quality","name":"Have a person read a sample every week","level":"Recommended","applies":["ai-features"],"rule":"Have a person read a random sample of production outputs every week, such as 30 per feature. Record each as good or bad with a reason.","check":"The review log has an entry for each AI feature in each of the last four weeks, with at least the set sample size.","url":"https://agentwranglers.se/playbook/ai-features/evals-and-quality/weekly-output-review","related":["/playbook/ai-features/evals-and-quality/evals-from-production"],"sources":[]},{"chapter":"Evals and quality","name":"Test that answers do not change with who asks","level":"Recommended","applies":["ai-features"],"rule":"For features that rank, score or decide about people, add eval pairs that differ only in one personal detail, such as a name or age. Fail the run when outcomes differ.","check":"Each feature that ranks, scores or decides about people has at least 20 counterfactual pairs in its eval set. CI fails when more than 1 pair in 20 gives two different outcomes.","url":"https://agentwranglers.se/playbook/ai-features/evals-and-quality/counterfactual-evals","related":["/playbook/ai-features/evals-and-quality/repeat-eval-runs","/playbook/compliance/ai-act/ai-high-risk-check","/playbook/compliance/ai-act/ai-register"],"sources":[{"label":"NIST AI 600-1, Generative AI Profile, section 2.6 on harmful bias","url":"https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf"},{"label":"Regulation (EU) 2026/1744 adding AI Act Article 4a","url":"https://eur-lex.europa.eu/eli/reg/2026/1744/oj"}]},{"chapter":"Guardrails and human review","name":"Check input before it reaches the model","level":"Recommended","applies":["ai-features"],"rule":"Run cheap checks on every input before the model call. Limit length, reject empty or off-topic requests and mask personal data the task does not need.","check":"Unit tests send an input over the limit, an empty input and one with a test card number. The first two get a clear error with no model call. The third reaches the fake provider with the number masked.","url":"https://agentwranglers.se/playbook/ai-features/guardrails-and-human-review/input-checks","related":["/playbook/adversarial-testing/ai-agents-and-models/direct-prompt-injection","/playbook/adversarial-testing/ai-agents-and-models/private-data-to-models"],"sources":[]},{"chapter":"Guardrails and human review","name":"Check output before anyone sees it","level":"Recommended","applies":["ai-features"],"rule":"Run every answer through checks in code before it is shown, sent or stored. Block or replace answers that fail.","check":"Unit tests feed the output checks an answer with an unknown link, one with an email address and one promising a refund. Each is blocked and logged, and the user sees the safe reply.","url":"https://agentwranglers.se/playbook/ai-features/guardrails-and-human-review/output-checks","related":["/playbook/adversarial-testing/ai-agents-and-models/untrusted-model-output","/playbook/ai-features/latency-cost-and-fallbacks/stream-answers"],"sources":[]},{"chapter":"Guardrails and human review","name":"Handle refusals and cut-off answers","level":"Recommended","applies":["ai-features"],"rule":"Read the stop reason of every model response. Show a refusal or a cut-off answer as such, with a next step, and never as a complete answer.","check":"A unit test with a fake provider returns each stop reason in turn, plus one invented value. The gateway maps each one to the right value and the invented one to `unknown`. The UI test shows no cut-off answer without its incomplete label.","url":"https://agentwranglers.se/playbook/ai-features/guardrails-and-human-review/refusals-and-truncation","related":["/playbook/app-ui/data-views/error-in-place","/playbook/ai-features/latency-cost-and-fallbacks/accessible-streaming"],"sources":[]},{"chapter":"Guardrails and human review","name":"Let a person decide what matters","level":"Required","risk":"law","applies":["ai-features"],"rule":"Never let model output alone decide something with legal, financial, medical or similar effects on a person. A named person reviews it first, with the means to overrule it.","check":"For each decision on the list, a test tries to complete it with no reviewer. It fails. Every completed decision in the database has a reviewer ID and a time.","url":"https://agentwranglers.se/playbook/ai-features/guardrails-and-human-review/human-decides","related":["/playbook/adversarial-testing/ai-agents-and-models/human-approval","/playbook/compliance/ai-act/ai-high-risk-check","/playbook/compliance/online-platforms/platform-complaints"],"sources":[{"label":"GDPR Article 22 on automated decisions","url":"https://eur-lex.europa.eu/eli/reg/2016/679/oj"},{"label":"AI Act Article 14 on human oversight","url":"https://eur-lex.europa.eu/eli/reg/2024/1689/oj"}]},{"chapter":"Guardrails and human review","name":"Label AI output where people see it","level":"Required","risk":"law","applies":["ai-features","eu-users"],"rule":"Mark AI-written text as AI-written wherever it is shown. Store a flag with it, so the label follows the text into emails, exports and other pages.","check":"A browser test opens each screen that shows AI output and finds the label. A query finds no stored AI output whose flag is empty, and every row with the flag cleared has an approver.","url":"https://agentwranglers.se/playbook/ai-features/guardrails-and-human-review/label-ai-output","related":["/playbook/website/privacy/disclose-ai","/playbook/compliance/ai-act/ai-transparency-map"],"sources":[{"label":"AI Act Article 50 on transparency obligations","url":"https://eur-lex.europa.eu/eli/reg/2024/1689/oj"}]},{"chapter":"Latency, cost and fallbacks","name":"Stream long answers","level":"Recommended","applies":["ai-features"],"rule":"Stream any answer a person waits for that can take more than 2 seconds, and measure the time to the first token.","check":"The latency dashboard shows time to first token per feature. Its 95th percentile is under the first-token target in the feature's spec, such as 1.5 seconds.","url":"https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/stream-answers","related":["/playbook/api-design/requests-and-responses/streamed-responses","/playbook/ai-features/guardrails-and-human-review/output-checks","/playbook/app-ui/feedback-and-dialogs/long-action-progress"],"sources":[{"label":"HTML Standard on server-sent events","url":"https://html.spec.whatwg.org/multipage/server-sent-events.html"}]},{"chapter":"Latency, cost and fallbacks","name":"Time out every model call","level":"Required","risk":"outage","applies":["ai-features"],"rule":"Give every model call a total time limit and, when streaming, a limit on the silence between chunks. Retry only errors that may pass, and only before any output reached the user.","check":"A test with a fake provider that never answers returns a timeout error within the limit plus one second. A fake that answers 503 twice and then succeeds gives one answer after three calls, not more. A search finds every SDK client built with its own retries set to 0.","url":"https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/model-timeouts","related":["/playbook/api-design/reliability/retry-backoff","/playbook/api-design/reliability/timeouts","/playbook/operations/disaster-recovery/model-provider-failure","/playbook/ai-features/latency-cost-and-fallbacks/provider-fallback"],"sources":[{"label":"RFC 9110 HTTP Semantics on Retry-After","url":"https://www.rfc-editor.org/rfc/rfc9110#field.retry-after"}]},{"chapter":"Latency, cost and fallbacks","name":"Put the stable part of the prompt first","level":"Recommended","applies":["ai-features"],"rule":"Order every prompt from most stable to least stable, with instructions, tool definitions and shared documents first and user input last. Turn on the provider's prompt caching, and measure the hit rate.","check":"The share of input tokens served from cache is on the dashboard per feature. For a feature whose shared prefix is over 4,096 tokens, it is above half during busy hours. A test renders the prompt twice for two users and finds the same bytes up to the first user-specific text.","url":"https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/prompt-caching","related":["/playbook/ai-features/latency-cost-and-fallbacks/cost-per-call"],"sources":[]},{"chapter":"Latency, cost and fallbacks","name":"Cache whole answers for repeated inputs","level":"Optional","applies":["ai-features"],"rule":"Cache the full answer for features that give one right answer per input, such as classification or extraction. Key the cache on the prompt version, the model version and a hash of the input.","check":"A test calls the feature twice with the same input. The provider is called once. After the prompt version changes, the provider is called again.","url":"https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/answer-cache","related":["/playbook/data/caches-and-copies/cache-freshness"],"sources":[]},{"chapter":"Latency, cost and fallbacks","name":"Price every call and cap the spend","level":"Required","risk":"money","applies":["ai-features"],"rule":"Compute the cost of every model call from its token counts and record it with the feature and user. Enforce a daily budget per user and per feature.","check":"A test sets a test user's budget to one call and makes two. The second is refused without a provider call. The same test with the feature's budget set to one call gives the same refusal. The cost per feature per day appears on the dashboard.","url":"https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/cost-per-call","related":["/playbook/adversarial-testing/ai-agents-and-models/cost-and-loop-limits","/playbook/ai-features/latency-cost-and-fallbacks/prompt-caching","/playbook/operations/capacity-and-cost/cost-monitoring"],"sources":[{"label":"OpenTelemetry semantic conventions for generative AI","url":"https://github.com/open-telemetry/semantic-conventions-genai"}]},{"chapter":"Latency, cost and fallbacks","name":"Keep every request inside a token budget","level":"Recommended","applies":["ai-features"],"rule":"Give every request a set input token budget and an explicit output limit. Trim conversation history and retrieved text by a fixed order before the call, never by letting the provider cut or reject it.","check":"A test sends a conversation of 200 long turns. The gateway sends at most the budget, the instructions and the latest message are intact, and the log shows the trimmed count. A search finds no model call without an output limit.","url":"https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/context-budget","related":["/playbook/ai-assisted-development/context-and-memory/subagents-for-side-work","/playbook/adversarial-testing/abuse-and-limits/resource-limits"],"sources":[]},{"chapter":"Latency, cost and fallbacks","name":"Keep working when the provider fails","level":"Required","risk":"outage","applies":["ai-features"],"rule":"Decide for each AI feature what happens when the model provider is down, slow or out of quota, and test that path. Use a second model that passed the same evals, a version without AI, or a clear message, never a hang or a broken page.","check":"In staging, block the main provider's address. Each AI feature still answers or shows its non-AI version within its time limit, and the logs show the fallback in use.","url":"https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/provider-fallback","related":["/playbook/operations/disaster-recovery/provider-failure","/playbook/operations/disaster-recovery/model-provider-failure","/playbook/ai-features/latency-cost-and-fallbacks/model-timeouts"],"sources":[]},{"chapter":"Latency, cost and fallbacks","name":"Send work nobody waits for as a batch","level":"Optional","applies":["ai-features"],"rule":"Run large jobs that no person waits for, such as nightly tagging or re-summarising documents, through the provider's batch interface.","check":"The job's log shows it was sent as a batch, and its cost per item is below the live price for the same model.","url":"https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/batch-offline-work","related":["/playbook/api-design/reliability/long-running-jobs"],"sources":[]},{"chapter":"Latency, cost and fallbacks","name":"Make streamed answers work with a screen reader","level":"Required","risk":"law","applies":["ai-features"],"rule":"Announce a streamed answer to screen reader users once, when it ends, not piece by piece. Give every streaming answer a stop button that also cancels the model call.","check":"A browser test plays a fake stream of 200 chunks. The answer container has no `aria-live` or live role, and the status region changes once. Pressing stop ends the stream within 1 second, and the fake provider records the cancelled call.","url":"https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/accessible-streaming","related":["/playbook/website/accessibility/status-messages","/playbook/ai-features/guardrails-and-human-review/refusals-and-truncation"],"sources":[{"label":"WCAG 2.2 criterion 4.1.3 Status Messages","url":"https://www.w3.org/TR/WCAG22/#status-messages"},{"label":"WCAG 2.2 criterion 2.2.2 Pause, Stop, Hide","url":"https://www.w3.org/TR/WCAG22/#pause-stop-hide"},{"label":"WAI-ARIA 1.2 on aria-busy","url":"https://www.w3.org/TR/wai-aria-1.2/#aria-busy"}]},{"chapter":"Logs, privacy and feedback","name":"Log every model call","level":"Recommended","applies":["ai-features"],"rule":"Write one structured log line per model call. It holds the feature, request ID, prompt ID and version, model version, tokens, cost, latency, ending and outcome.","check":"Parse one hour of gateway logs with a JSON tool. Every `ai.call` line has all the listed fields, and none contains a field holding prompt or answer text.","url":"https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/log-model-calls","related":["/playbook/ai-features/prompts-as-code/prompt-version-in-logs","/playbook/ai-features/deciding-to-use-a-model/model-gateway","/playbook/operations/logs/request-ids"],"sources":[{"label":"OpenTelemetry semantic conventions for generative AI","url":"https://github.com/open-telemetry/semantic-conventions-genai"}]},{"chapter":"Logs, privacy and feedback","name":"Keep prompt and answer text briefly and apart","level":"Required","risk":"data","applies":["ai-features"],"rule":"Store the text of prompts and answers apart from other logs, with personal data masked and access limited to named people. Delete it automatically after a set time, such as 30 days.","check":"The oldest row in the prompt text store is younger than the set limit. A search of it for a test user's email address finds only masked values. The access list holds only named people, with no group or shared account.","url":"https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/prompt-text-retention","related":["/playbook/data/personal-data/retention-jobs","/playbook/operations/logs/logs-without-personal-data","/playbook/ai-features/agents-in-products/agent-run-trace"],"sources":[{"label":"GDPR Article 5(1)(e) on storage limitation","url":"https://eur-lex.europa.eu/eli/reg/2016/679/oj"}]},{"chapter":"Logs, privacy and feedback","name":"Collect feedback on every answer","level":"Recommended","applies":["ai-features"],"rule":"Put a thumbs up, thumbs down and optional reason next to every AI answer, and store it with the call ID. Triage thumbs-down items every week.","check":"A browser test clicks thumbs down on an answer and picks a reason. A row with that call ID and reason appears. The triage log has an entry for each of the last four weeks.","url":"https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/feedback-on-every-answer","related":["/playbook/ai-features/deciding-to-use-a-model/design-for-wrong-answers","/playbook/ai-features/evals-and-quality/evals-from-production","/playbook/data/product-analytics/tracking-plan"],"sources":[]},{"chapter":"Logs, privacy and feedback","name":"Train only on data people agreed to share","level":"Required","risk":"law","applies":["ai-features"],"rule":"Use customer prompts, answers or documents to train a model only with a recorded legal basis, such as consent. Never send them to a provider that trains on them by default.","check":"The script that builds training sets selects only rows with the flag on. A test with one opted-in and one opted-out user produces a set with only the first user's data. The terms of each provider plan in use say it does not train on your data.","url":"https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/consent-before-training","related":["/playbook/adversarial-testing/ai-agents-and-models/private-data-to-models"],"sources":[{"label":"GDPR Article 6 on lawful processing","url":"https://eur-lex.europa.eu/eli/reg/2016/679/oj"}]},{"chapter":"Logs, privacy and feedback","name":"Erase a person's data from every AI store","level":"Required","risk":"law","applies":["ai-features"],"rule":"When a person's data is deleted, delete it from every store an AI feature fills. That includes vectors, prompt text, answer caches, agent memory, run traces and files kept by the model provider.","check":"Create a test person, use every AI feature as them, then delete them. A search of each AI store and a list call to each provider store find no row or file with their ID. A search of the code finds no Responses API call without an explicit `store` value.","url":"https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/erase-from-ai-stores","related":["/playbook/compliance/records-and-data-map/data-map","/playbook/website/privacy/data-requests"],"sources":[{"label":"GDPR Article 17 on the right to erasure","url":"https://eur-lex.europa.eu/eli/reg/2016/679/oj"}]},{"chapter":"Agents in products","name":"Use a fixed workflow when the steps are known","level":"Recommended","applies":["ai-features"],"rule":"Build a fixed sequence of model calls and code when the steps of a task are known in advance. Use an agent loop, where the model picks the next tool, only when they are not.","check":"Each agent loop in the code has a decision note that lists the tasks it handles and why fixed steps could not. A search for agent loops finds none without a note.","url":"https://agentwranglers.se/playbook/ai-features/agents-in-products/workflow-before-agent","related":["/playbook/adversarial-testing/ai-agents-and-models/cost-and-loop-limits"],"sources":[]},{"chapter":"Agents in products","name":"End every agent run with a known status","level":"Recommended","applies":["ai-features"],"rule":"End every agent run in one of a fixed set of statuses and show it to the user. Examples are done, waiting for approval, approval expired, step limit, time limit, budget limit and failed.","check":"Tests run the agent against four fake models. One never finishes, one runs past the time limit, one spends past the budget and one asks for approval. Each run ends with the matching status, and the user sees a message for each.","url":"https://agentwranglers.se/playbook/ai-features/agents-in-products/agent-run-status","related":["/playbook/architecture/data-shapes/states-as-types"],"sources":[]},{"chapter":"Agents in products","name":"Record every step of every run","level":"Recommended","applies":["ai-features"],"rule":"Store each step of an agent run with its tool name, arguments, a short result, the time and the cost. Let support and the user see what the agent did.","check":"Run a test task with three tool calls. The run's trace has three step rows with tool, arguments, result, time and cost. The user's activity page lists the same actions.","url":"https://agentwranglers.se/playbook/ai-features/agents-in-products/agent-run-trace","related":["/playbook/ai-features/logs-privacy-and-feedback/prompt-text-retention","/playbook/adversarial-testing/hostile-agents-and-bots/reversible-agent-actions"],"sources":[]},{"chapter":"Agents in products","name":"Let runs pause and resume","level":"Recommended","applies":["ai-features"],"rule":"Save an agent run's state after every step, so it can wait for approval and survive a restart. Expire pending approvals after a set time, such as 24 hours.","check":"Start a run that needs approval, restart the service, then approve. The run finishes from where it stopped. An approval older than the limit is refused with an expired message.","url":"https://agentwranglers.se/playbook/ai-features/agents-in-products/resumable-agent-runs","related":["/playbook/adversarial-testing/ai-agents-and-models/human-approval"],"sources":[]},{"chapter":"Agents in products","name":"Test agents by the end state","level":"Recommended","applies":["ai-features"],"rule":"Test an agent with scenarios that check the final state and the actions it must never take, not the exact sequence of steps.","check":"CI runs the scenario set on every prompt, tool or model change. It fails when a scenario misses its end state or calls a forbidden tool in any run.","url":"https://agentwranglers.se/playbook/ai-features/agents-in-products/agent-scenario-tests","related":["/playbook/ai-features/evals-and-quality/repeat-eval-runs","/playbook/testing/unit-tests/unit-behaviour"],"sources":[]}]}