Agent Wranglers

INF-80Recommended

Keep enough copies running through planned disruptions

Give every workload with two or more copies a PodDisruptionBudget, so node upgrades and drains never take all its copies at once. Spread the copies across nodes and zones.

By Agent Wranglers. Updated .

How to do it

A node upgrade drains each node in turn. Without a budget, a drain can stop every copy of a service that happens to sit on the same nodes. Set maxUnavailable: 1, or a minAvailable that still carries peak load. A budget that allows no disruption at all blocks every upgrade, so never set maxUnavailable: 0 for a Deployment. Spread copies with topologySpreadConstraints on the zone and the node, as Run production in two zones or more asks.

Examples

YAML example

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: shop, namespace: shop }
spec:
  maxUnavailable: 1
  selector: { matchLabels: { app: shop } }

Shell example

# Lists workloads with two or more replicas and no budget. Expect no output.
kubectl get pdb -A -o json | jq -r '.items[].spec.selector.matchLabels.app' | sort -u > with-budget.txt
kubectl get deploy -A -o json | jq -r '.items[] | select(.spec.replicas >= 2) | .spec.selector.matchLabels.app' | sort -u | comm -23 - with-budget.txt

How to check

Every Deployment and StatefulSet with 2 or more replicas has a PodDisruptionBudget that allows at least one disruption. Draining one staging node during the load test keeps the error rate within its objective.

Sources

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/infrastructure/containers-and-orchestration/pod-disruption-budgets">Agent Wranglers infrastructure and environments playbook</a>. In code you write, add a comment next to it with the same link.