Probe every Kubernetes workload for readiness and liveness

How to do it

Without a readiness probe, Kubernetes sends traffic to a pod as soon as its process starts, so every deploy drops requests. A liveness probe that calls the database restarts every pod when the database is slow, which turns a slow database into a full outage. So the liveness probe asks only whether the process answers, and the readiness probe checks what the pod needs to serve. A startup probe holds the other probes back until the app has started, so a slow start is not killed in a loop. The two checks are defined in Separate alive from ready. Plain Docker reads the image's own check instead, as Declare a health check in the image says.

Examples

YAML example

containers:
  - name: shop
    image: registry.example.com/shop:1.42.0@sha256:<digest>
    ports: [{ containerPort: 8080 }]
    startupProbe:
      httpGet: { path: /healthz, port: 8080 }
      failureThreshold: 30
      periodSeconds: 2
    livenessProbe:
      httpGet: { path: /healthz, port: 8080 }
      periodSeconds: 10
    readinessProbe:
      httpGet: { path: /readyz, port: 8080 }
      periodSeconds: 5

Shell example

# Lists every container in the cluster without a readiness or liveness probe. Expect no output.
kubectl get deploy,statefulset -A -o json | jq -r '.items[] | .metadata.namespace + "/" + .metadata.name as $n
  | .spec.template.spec.containers[] | select(.readinessProbe == null or .livenessProbe == null) | $n + " " + .name'

How to check

A policy check on the manifests fails any Deployment or StatefulSet container with a port and no readiness or liveness probe. On staging, stopping the database marks pods not ready, and no pod restarts.

Sources

• Kubernetes, Liveness, readiness and startup probes

Related rules

• ENG-9 Give the server a health check in Engineering
• OPS-59 Declare a health check in the image in Operations and observability