Agent Wranglers

Recommended

Test what a failover loses

When a database has a standby for failover, write down how much data a failover may lose. Then promote the standby in a drill.

How to do it

PostgreSQL replication is asynchronous by default, so commits not yet copied are lost when the primary fails. The loss grows with replication lag. Synchronous replication makes each commit wait for the standby, but commits stop when the only synchronous standby fails. Choose between them from the data loss limit in Set recovery targets. PostgreSQL does not detect a failed primary by itself, so use a managed service or a tool such as Patroni. Make sure the old primary can never take writes again, because two primaries lose data. Alert on lag as Read from replicas only what may lag does.

Examples

Using these examples: they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

SQL example

-- On the primary: how far each standby is behind, in bytes
SELECT application_name, state, sync_state,
       pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_lag_bytes
FROM pg_stat_replication;
-- In the drill, on the standby
SELECT pg_promote();

INI example

# postgresql.conf: each commit waits for one of two standbys
synchronous_standby_names = 'ANY 1 (standby_a, standby_b)'
synchronous_commit = on

How to check

A drill on staging promotes the standby under write load. The app reconnects within the recovery time objective, the old primary refuses writes, and the lost writes fit the recovery point objective.

Sources

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/data/backups-and-recovery/failover">Agent Wranglers data playbook</a>. In code you write, add a comment next to it with the same link.