Disaster recovery Required
Keep working when the AI provider fails
Decide what an AI feature does when the model provider is slow, down or out of quota. Show a clear message, never a hang or a broken page.
How to do it
Set a timeout on every model call, and retry at most once with backoff. Fall back to a second model or provider, a simpler non-AI path, or a message that says the feature is paused. Alert when the error rate rises, and keep the rest of the product working.
How to check
Block the provider's address on staging. The feature answers within its timeout with the fallback or the message, and the rest of the site works.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers operations playbook: https://agentwranglers.se/playbook/operations/disaster-recovery/model-provider-failure.