Servers

These rules cover virtual machines and rented servers, from a fleet behind a load balancer to one small machine that runs everything. Containers have their own chapter in the operations playbook. How each service runs on the machine is covered in Run each service as its own user and Lock the service in a sandbox.

Required

• Configure long-lived servers from code

  Required

  When a server lives for months, set up its packages, files and services from code that can run again without changing anything.

• Rebuild any server from code

  Required

  Keep everything needed to rebuild each server in code, so one command builds a fresh one. Prove it on a new machine in the quarterly restore drill.

• Install security updates by themselves

  Required

  Let every server install security updates by itself every day, and reboot in a set window when the kernel needs it.

• Turn on a firewall on every host

  Required

  Run a firewall on every server that refuses all incoming traffic except the ports it serves. Allow SSH only from named admin addresses.

• Allow SSH with keys only

  Required

  Turn off password logins and direct root logins over SSH. Give each person their own key, and remove it the day they leave.

Recommended

• Replace servers instead of patching them

  Recommended

  Build a new machine image for every change, start new servers from it and stop the old ones. Never change a running server by hand.