Stop calling a failing service with a circuit breaker

How to do it

Without a breaker, every request waits its full timeout on a service that is down, and the waiting requests fill the workers. A breaker has three states. Closed passes calls and counts failures. Open fails at once without a network call. Half-open lets a few trial calls through and closes again when they succeed. Open it on a failure rate over enough calls, such as 50% of at least 20 calls. Then two errors at night do not trip it. When it is open, take the fallback chosen in Decide what happens when a provider is down. Libraries exist for most languages, such as opossum for Node.js, pybreaker for Python, Resilience4j for Java and Polly for .NET. Report each breaker's state as a metric, and alert when one stays open for 5 minutes. The breaker stops calls to one service. Cap retries across the whole service limits retries. Turn work away before overload protects your own service. Keep the app working when the cache is gone covers the cache.

Examples

JavaScript example

import CircuitBreaker from 'opossum';

const shippingBreaker = new CircuitBreaker(fetchShippingRates, {
  timeout: 3000,
  errorThresholdPercentage: 50,
  volumeThreshold: 20,
  resetTimeout: 30_000,
});
shippingBreaker.fallback(order => cachedShippingRates(order));
shippingBreaker.on('open', () => breakerOpen.set({ dependency: 'shipping' }, 1));
shippingBreaker.on('close', () => breakerOpen.set({ dependency: 'shipping' }, 0));

export const shippingRates = order => shippingBreaker.fire(order);

Python example

import pybreaker

shipping_breaker = pybreaker.CircuitBreaker(fail_max=5, reset_timeout=30, name="shipping")


def shipping_rates(order):
    try:
        return shipping_breaker.call(fetch_shipping_rates, order)
    except pybreaker.CircuitBreakerError:
        return cached_shipping_rates(order)

How to check

Block one dependency on staging. Within 30 seconds, calls to it fail in under 10 ms with no network traffic, the fallback answers, and the metric shows the breaker open. Unblock it. The breaker closes within one pause plus the trial calls.

Sources

• AWS Prescriptive Guidance, Circuit breaker pattern

Related rules

• API-78 Decide what happens when a provider is down in API design
• OPS-66 Cap retries across the whole service in Operations and observability
• OPS-40 Turn work away before overload in Operations and observability
• SCL-15 Keep the app working when the cache is gone in Scaling