Agent Wranglers

Recommended

Keep server clocks in sync

Run a time sync client on every server with at least two time sources. Keep every server on UTC, and alert when a clock is more than 1 second off.

By Agent Wranglers. Updated .

How to do it

Clocks drift. A wrong clock breaks the order of logs across servers, certificate checks, signed tokens and one-time codes. Use the system's client, such as chrony or systemd-timesyncd. Point it at your cloud's time service or a public pool, with two sources at least. Set the time zone to UTC, so logs from different machines line up as Log structured events expects.

Examples

Using these examples: they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

Shell example

timedatectl set-timezone Etc/UTC
timedatectl show -p NTPSynchronized --value   # yes
chronyc tracking | grep 'System time'         # System time : 0.000012 seconds fast of NTP time
chronyc sources                               # two or more sources, one marked *

Plain text example

# /etc/chrony/chrony.conf
pool 2.pool.ntp.org iburst maxsources 2
server time.example.com iburst
makestep 1 3

How to check

On every server, timedatectl show -p NTPSynchronized --value prints yes and timedatectl show -p Timezone --value prints Etc/UTC or UTC. chronyc tracking shows a system time offset under 1 second.

Sources

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/infrastructure/servers/time-sync">Agent Wranglers infrastructure playbook</a>. In code you write, add a comment next to it with the same link.