---
title: "Cap retries across the whole service · Agent Wranglers"
url: https://agentwranglers.se/playbook/operations/capacity-and-cost/retry-budget
updated: 2026-10-21
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Cap retries across the whole service

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Capacity and cost](https://agentwranglers.se/playbook/operations/capacity-and-cost.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers). Updated 2026-10-21.

- ID: OPS-66
- Level: Recommended
- Chapter: [Capacity and cost](https://agentwranglers.se/playbook/operations/capacity-and-cost)
- Rule: Retry a failed call at one layer only, and cap retries in each process at 10% of its calls to that service. Pass the caller's deadline on with every call.
- How: When three layers each try 3 times, the lowest service gets 27 calls for one request. That flood keeps a struggling service down after the first fault is gone. Choose the one layer that retries, usually the one nearest the user, and let inner layers fail fast. Count calls and retries in each process over a rolling minute. When retries pass the budget, return the error instead of trying again. Send the time left with each call, for example as a `deadline` header, and skip the call when the time is gone. The backoff and the codes worth retrying are in [retry with backoff](https://agentwranglers.se/playbook/api-design/reliability/retry-backoff), and the time limits in [timeouts on every call](https://agentwranglers.se/playbook/api-design/reliability/timeouts). Shed what still gets through with [turn work away before overload](https://agentwranglers.se/playbook/operations/capacity-and-cost/load-shedding).
- Check: Make a dependency on staging return 503 for every call for 5 minutes. Calls to it stay at or under 1.1 times the normal call rate, and only one layer shows retries in the traces.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### JavaScript example

```js
const RETRY_SHARE = 0.1;
const WINDOW_MS = 60_000;
let calls = 0;
let retries = 0;
setInterval(() => { calls = 0; retries = 0; }, WINDOW_MS);

export function mayRetry() {
  if (retries >= calls * RETRY_SHARE) return false;
  retries += 1;
  return true;
}

export function countCall() { calls += 1; }
```

### Python example

```python
import threading

RETRY_SHARE = 0.1
WINDOW_SECONDS = 60

class RetryBudget:
    def __init__(self):
        self.calls = 0
        self.retries = 0
        self.lock = threading.Lock()

    def count_call(self):
        with self.lock:
            self.calls += 1

    def may_retry(self):
        with self.lock:
            if self.retries >= self.calls * RETRY_SHARE:
                return False
            self.retries += 1
            return True

    def reset(self):
        with self.lock:
            self.calls = self.retries = 0
```

## Sources

- [Google SRE book, Addressing Cascading Failures](https://sre.google/sre-book/addressing-cascading-failures/)

## Related rules

- [Send the same idempotency key when you retry a write](https://agentwranglers.se/playbook/api-design/using-other-companies-apis/outgoing-idempotency-keys) in API design
- [Retry with backoff and jitter](https://agentwranglers.se/playbook/api-design/reliability/retry-backoff) in API design
- [Set timeouts on every call](https://agentwranglers.se/playbook/api-design/reliability/timeouts) in API design
- [Turn work away before overload](https://agentwranglers.se/playbook/operations/capacity-and-cost/load-shedding) in Operations

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/operations/capacity-and-cost/retry-budget">Agent Wranglers operations playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Turn work away before overload](https://agentwranglers.se/playbook/operations/capacity-and-cost/load-shedding)
- Next: [Size the runtime's memory to the box](https://agentwranglers.se/playbook/operations/capacity-and-cost/runtime-memory-limits)
- [Capacity and cost](https://agentwranglers.se/playbook/operations/capacity-and-cost)