Keep working when the provider fails

How to do it

Choose one fallback per feature. Options are a second model from another provider or region, a smaller model, a cached answer or the feature without AI, such as plain search. Run the eval set on the fallback model too, because it will answer differently. Switch after a set number of failures in a row, such as 5. Try the main model again after a pause, such as 60 seconds. Show users a note when the fallback is in use. Retries before the switch follow model timeouts. The operations side, alerts and the rest of the product, is in AI provider failure.

Example

export async function generateWithFallback<T>(request: GenerateRequest<T>): Promise<T & { servedBy: string }> {
  if (!breaker.isOpen('primary')) {
    try { return { ...(await primary.generate(request)), servedBy: 'primary' }; }
    catch (error) { breaker.recordFailure('primary'); log.warn({ event: 'ai.primary_failed', error: String(error) }); }
  }
  return { ...(await fallback.generate(request)), servedBy: 'fallback' };
}

def generate_with_fallback(request: GenerateRequest) -> Result:
    if not breaker.is_open("primary"):
        try:
            return primary.generate(request).with_served_by("primary")
        except ProviderError as error:
            breaker.record_failure("primary")
            log.warning("ai.primary_failed", error=str(error))
    return fallback.generate(request).with_served_by("fallback")

How to check

In staging, block the main provider's address. Each AI feature still answers or shows its non-AI version within its time limit, and the logs show the fallback in use.