Answer with a fallback when the model fails or is slow

How to do it

A model call is a network call with a heavy computation behind it. Set the timeout from the latency budget of the page or job. The fallback is the simple baseline or the safe choice, such as sending a refund to manual review. Log each fallback with its reason and alert when the share rises past a limit. The same rule for hosted language models is Keep working when the provider fails.

Examples

Python example

import concurrent.futures

MODEL_TIMEOUT_SECONDS = 0.15
pool = concurrent.futures.ThreadPoolExecutor(max_workers=8)

def refund_decision(features: dict, model, baseline, log) -> str:
    try:
        score = pool.submit(model.predict, features).result(timeout=MODEL_TIMEOUT_SECONDS)
        return "review" if score >= model.threshold else "pay"
    except Exception as error:
        log("model_fallback", reason=type(error).__name__)
        return baseline(features)

TypeScript example

const MODEL_TIMEOUT_MS = 150;

export async function refundDecision(features: object, predict: (f: object, signal: AbortSignal) => Promise<number>, threshold: number,
  baseline: (f: object) => 'review' | 'pay', log: (event: string, reason: string) => void) {
  try {
    const score = await predict(features, AbortSignal.timeout(MODEL_TIMEOUT_MS));
    return score >= threshold ? 'review' : 'pay';
  } catch (error) {
    log('model_fallback', (error as Error).name);
    return baseline(features);
  }
}

How to check

A test stops the model server and sends requests. Each one gets the fallback within the timeout plus 100 ms, and the fallback count rises by the number of requests.

Sources

• Google SRE book, Addressing cascading failures

Related rules

• MLO-2 Beat a simple baseline before you ship a model in Machine learning
• AIF-44 Keep working when the provider fails in AI features