Capacity and resilience

Required

• Size the runtime's memory to the box

  OPS-41

  When the language runtime manages memory itself, as Java, Node, .NET and Go do, cap its heap below the container or service limit. It then frees memory before the system kills it.

• Warn before resources run out

  OPS-42

  Alert on how soon a disk, memory pool or quota will be full, not only on how full it is.

Recommended

• Keep headroom

  OPS-39

  Keep normal peak load below a set share of the tested limit, such as 60 percent. Then a spike or a lost server does not break the service.

• Turn work away before overload

  OPS-40

  Cap how many requests each instance handles at once. Answer the rest at once with 503 and a Retry-After header instead of queueing them.

• Match timeouts from proxy to app and stop abandoned work

  OPS-70

  Give each layer a timeout a few seconds shorter than the layer in front of it. That runs from the load balancer to the proxy, the app and the database. Stop work as soon as the client or the proxy gives up on it.

• Cap retries across the whole service

  OPS-66

  Retry a failed call at one layer only, and cap retries in each process at 10% of its calls to that service. Pass the caller's deadline on with every call.

• Stop calling a failing service with a circuit breaker

  OPS-74

  Wrap each call to another service in a circuit breaker. After a set share of failures it opens and fails at once for a pause, such as 30 seconds. Then it lets trial calls through before it closes.

• Run production in two zones or more

  OPS-78

  Run every production service and data store in at least two availability zones. Losing one zone must leave the service within its objectives.

Read this with

• Scaling playbook

  Watch saturation and spend in production, and add capacity before a limit is reached.