Retry a bad answer and then fall back

How to do it

Catch the validation error in the gateway. Send the same request again with a short note naming the failed field. After the last retry, return a fallback the feature can handle, such as "unknown", a fixed message or a hand-off to a person. Count failures per prompt version and alert when more than 2 in 100 calls fail, because that points to a prompt or model problem.

Example

const MAX_ATTEMPTS = 3;

export async function generateValid<T>(call: (hint?: string) => Promise<string>, schema: ZodType<T>, fallback: T): Promise<T> {
  let hint: string | undefined;
  for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
    const parsed = schema.safeParse(parseJsonOrUndefined(await call(hint)));
    if (parsed.success) return parsed.data;
    hint = `Your last answer was invalid: ${parsed.error.issues.map(i => `${i.path.join('.')} ${i.message}`).join(', ')}`;
  }
  metrics.increment('ai.invalid_output');
  return fallback;
}

MAX_ATTEMPTS = 3

def generate_valid(call: Callable[[str | None], str], schema: type[T], fallback: T) -> T:
    hint = None
    for _ in range(MAX_ATTEMPTS):
        try:
            return schema.model_validate_json(call(hint))
        except ValidationError as error:
            hint = f"Your last answer was invalid: {error.errors(include_input=False)}"
    metrics.increment("ai.invalid_output")
    return fallback

How to check

A unit test with a fake provider returns invalid JSON three times. The gateway makes exactly three calls and returns the fallback. The failure counter shows one failed request.