Latency, cost and fallbacks

Required

• Time out every model call

  Required

  Give every model call a total time limit and, when streaming, a limit on the silence between chunks. Retry only errors that may pass, and only before any output reached the user.

• Price every call and cap the spend

  Required

  Compute the cost of every model call from its token counts and record it with the feature and user. Enforce a daily budget per user and per feature.

• Keep every request inside a token budget

  Required

  Give every request a set input token budget and an explicit output limit. Trim conversation history and retrieved text by a fixed order before the call, never by letting the provider cut or reject it.

• Keep working when the provider fails

  Required

  Decide for each AI feature what happens when the model provider is down, slow or out of quota, and test that path. Use a second model that passed the same evals, a version without AI, or a clear message, never a hang or a broken page.

Recommended

• Stream long answers

  Recommended

  Stream any answer a person waits for that can take more than 2 seconds, and measure the time to the first token.

• Put the stable part of the prompt first

  Recommended

  Order every prompt from most stable to least stable, with instructions, tool definitions and shared documents first and user input last. Turn on the provider's prompt caching, and measure the hit rate.

Optional

• Cache whole answers for repeated inputs

  Optional

  Cache the full answer for features that give one right answer per input, such as classification or extraction. Key the cache on the prompt version, the model version and a hash of the input.

• Send work nobody waits for as a batch

  Optional

  Run large jobs that no person waits for, such as nightly tagging or re-summarising documents, through the provider's batch interface.