Cap retries across the whole service

How to do it

When three layers each try 3 times, the lowest service gets 27 calls for one request. That flood keeps a struggling service down after the first fault is gone. Choose the one layer that retries, usually the one nearest the user, and let inner layers fail fast. Count calls and retries in each process over a rolling minute. When retries pass the budget, return the error instead of trying again. Send the time left with each call, for example as a deadline header, and skip the call when the time is gone. The backoff and the codes worth retrying are in retry with backoff, and the time limits in timeouts on every call. Shed what still gets through with turn work away before overload.

Examples

JavaScript example

const RETRY_SHARE = 0.1;
const WINDOW_MS = 60_000;
let calls = 0;
let retries = 0;
setInterval(() => { calls = 0; retries = 0; }, WINDOW_MS);

export function mayRetry() {
  if (retries >= calls * RETRY_SHARE) return false;
  retries += 1;
  return true;
}

export function countCall() { calls += 1; }

Python example

import threading

RETRY_SHARE = 0.1
WINDOW_SECONDS = 60

class RetryBudget:
    def __init__(self):
        self.calls = 0
        self.retries = 0
        self.lock = threading.Lock()

    def count_call(self):
        with self.lock:
            self.calls += 1

    def may_retry(self):
        with self.lock:
            if self.retries >= self.calls * RETRY_SHARE:
                return False
            self.retries += 1
            return True

    def reset(self):
        with self.lock:
            self.calls = self.retries = 0

How to check

Make a dependency on staging return 503 for every call for 5 minutes. Calls to it stay at or under 1.1 times the normal call rate, and only one layer shows retries in the traces.

Sources

• Google SRE book, Addressing Cascading Failures

Related rules

• Send the same idempotency key when you retry a write in API design
• Retry with backoff and jitter in API design
• Set timeouts on every call in API design
• Turn work away before overload in Operations