---
title: "Cap retries across the whole service · Agent Wranglers"
url: https://agentwranglers.se/playbook/operations/capacity-and-resilience/retry-budget
id: OPS-66
level: Recommended
updated: 2026-10-27
build: a2a6bdfed830
tokens: 1188
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Cap retries across the whole service

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Capacity and resilience](https://agentwranglers.se/playbook/operations/capacity-and-resilience.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: OPS-66
- Level: Recommended
- Chapter: [Capacity and resilience](https://agentwranglers.se/playbook/operations/capacity-and-resilience.md)
- Rule: Retry a failed call at one layer only, and cap retries in each process at 10% of its calls to that service. Pass the caller's deadline on with every call.
- Check: Make a dependency on staging return 503 for every call for 5 minutes. Calls to it stay at or under 1.1 times the normal call rate, and only one layer shows retries in the traces.
- How: When three layers each try 3 times, the lowest service gets 27 calls for one request. That flood keeps a struggling service down after the first fault is gone. Choose the one layer that retries, usually the one nearest the user, and let inner layers fail fast. Count calls and retries in each process over a rolling minute. When retries pass the budget, return the error instead of trying again. Send the time left with each call, for example as a `deadline` header, and skip the call when the time is gone. The backoff and the codes worth retrying are in [retry with backoff](https://agentwranglers.se/playbook/api-design/reliability/retry-backoff.md), and the time limits in [timeouts on every call](https://agentwranglers.se/playbook/api-design/reliability/timeouts.md). Shed what still gets through with [turn work away before overload](https://agentwranglers.se/playbook/operations/capacity-and-resilience/load-shedding.md).

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### JavaScript example

```js
const RETRY_SHARE = 0.1;
const WINDOW_MS = 60_000;
let calls = 0;
let retries = 0;
setInterval(() => { calls = 0; retries = 0; }, WINDOW_MS);

export function mayRetry() {
  if (retries >= calls * RETRY_SHARE) return false;
  retries += 1;
  return true;
}

export function countCall() { calls += 1; }
```

### Python example

```python
import threading

RETRY_SHARE = 0.1
WINDOW_SECONDS = 60

class RetryBudget:
    def __init__(self):
        self.calls = 0
        self.retries = 0
        self.lock = threading.Lock()

    def count_call(self):
        with self.lock:
            self.calls += 1

    def may_retry(self):
        with self.lock:
            if self.retries >= self.calls * RETRY_SHARE:
                return False
            self.retries += 1
            return True

    def reset(self):
        with self.lock:
            self.calls = self.retries = 0
```

## Sources

- [Google SRE book, Addressing Cascading Failures](https://sre.google/sre-book/addressing-cascading-failures/)
- [Google SRE book, Handling Overload](https://sre.google/sre-book/handling-overload/)

## Related rules

- API-76 [Send the same idempotency key when you retry a write](https://agentwranglers.se/playbook/api-design/using-other-companies-apis/outgoing-idempotency-keys.md) in API design
- API-34 [Retry with backoff and jitter](https://agentwranglers.se/playbook/api-design/reliability/retry-backoff.md) in API design
- API-35 [Set timeouts on every call](https://agentwranglers.se/playbook/api-design/reliability/timeouts.md) in API design
- OPS-40 [Turn work away before overload](https://agentwranglers.se/playbook/operations/capacity-and-resilience/load-shedding.md) in Operations and observability
- OPS-74 [Stop calling a failing service with a circuit breaker](https://agentwranglers.se/playbook/operations/capacity-and-resilience/circuit-breakers.md) in Operations and observability

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/operations/capacity-and-resilience/retry-budget">Agent Wranglers operations and observability playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Match timeouts from proxy to app and stop abandoned work](https://agentwranglers.se/playbook/operations/capacity-and-resilience/timeout-chain.md)
- Next: [Stop calling a failing service with a circuit breaker](https://agentwranglers.se/playbook/operations/capacity-and-resilience/circuit-breakers.md)
- [Capacity and resilience](https://agentwranglers.se/playbook/operations/capacity-and-resilience.md)