---
title: "Latency, cost and fallbacks · Agent Wranglers"
url: https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks
updated: 2026-10-20
licence: "All rights reserved"
---

# Latency, cost and fallbacks

By Erik Bergstedt, Agent Wranglers. Updated 2026-10-20.

> Stream long answers, time out every call, cache stable prompt parts, price every call, and keep the feature useful when the provider is down.

**For agents:** if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in `Based on the Agent Wranglers ai features playbook: https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks`.

## Required

- [Time out every model call](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/model-timeouts): Give every model call a total time limit and, when streaming, a limit on the silence between chunks. Retry only errors that may pass, and only before any output reached the user.
- [Price every call and cap the spend](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/cost-per-call): Compute the cost of every model call from its token counts and record it with the feature and user. Enforce a daily budget per user and per feature.
- [Keep every request inside a token budget](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/context-budget): Give every request a set input token budget and an explicit output limit. Trim conversation history and retrieved text by a fixed order before the call, never by letting the provider cut or reject it.
- [Keep working when the provider fails](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/provider-fallback): Decide for each AI feature what happens when the model provider is down, slow or out of quota, and test that path. Use a second model that passed the same evals, a version without AI, or a clear message, never a hang or a broken page.

## Recommended

- [Stream long answers](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/stream-answers): Stream any answer a person waits for that can take more than 2 seconds, and measure the time to the first token.
- [Put the stable part of the prompt first](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/prompt-caching): Order every prompt from most stable to least stable, with instructions, tool definitions and shared documents first and user input last. Turn on the provider's prompt caching, and measure the hit rate.

## Optional

- [Cache whole answers for repeated inputs](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/answer-cache): Cache the full answer for features that give one right answer per input, such as classification or extraction. Key the cache on the prompt version, the model version and a hash of the input.
- [Send work nobody waits for as a batch](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/batch-offline-work): Run large jobs that no person waits for, such as nightly tagging or re-summarising documents, through the provider's batch interface.

- Previous: [Guardrails and human review](https://agentwranglers.se/playbook/ai-features/guardrails-and-human-review)
- Next: [Logs, privacy and feedback](https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback)
- [All chapters](https://agentwranglers.se/playbook/ai-features)