Give each outside dependency its own pool

How to do it

With one shared pool, a provider that hangs holds every connection, and calls to healthy services wait too. This is the bulkhead pattern, named after the walls in a ship's hull. Size each pool from the calls at once that the service needs at peak. Combine it with Set timeouts on every call and a circuit breaker. Slow background work gets its own workers in the same way, so it cannot take the threads that serve pages.

Examples

TypeScript example

// Each provider gets its own limit on calls at once
import pLimit from 'p-limit';
const shippingCalls = pLimit(10);
const paymentCalls = pLimit(20);

const quote = (parcel: Parcel) => shippingCalls(() => shippingApi.quote(parcel));
const charge = (order: Order) => paymentCalls(() => paymentApi.charge(order));

Python example

# Each provider gets its own connection pool
shipping = httpx.AsyncClient(base_url="https://shipping.example.com",
                             limits=httpx.Limits(max_connections=10), timeout=2.0)
payments = httpx.AsyncClient(base_url="https://payments.example.com",
                             limits=httpx.Limits(max_connections=20), timeout=5.0)

How to check

A test makes one provider hang and sends traffic that needs a second provider. Calls to the second provider still finish within their normal time.

Sources

• Microsoft: Bulkhead pattern

Related rules

• API-35 Set timeouts on every call in API design
• OPS-74 Stop calling a failing service with a circuit breaker in Operations and observability