Run each scheduled job once at a time

How to do it

With two servers, a slow run or a failover, the same timer fires twice and customers get two invoices. Skip a run rather than risk running it twice, since a skipped run is easier to repair. Use one lock that every server sees, such as a database advisory lock, or the scheduler's own setting, such as concurrencyPolicy: Forbid in Kubernetes. A lock can still fail during a failover, so record each item a job handles by its key, as in make writes safe to repeat. Catch skipped runs with job heartbeats.

Examples

JavaScript example

const JOB_LOCK_ID = 4201;
const { rows } = await db.query('SELECT pg_try_advisory_lock($1) AS locked', [JOB_LOCK_ID]);
if (!rows[0].locked) process.exit(0);
try {
  await sendMonthlyInvoices();
} finally {
  await db.query('SELECT pg_advisory_unlock($1)', [JOB_LOCK_ID]);
}

Python example

JOB_LOCK_ID = 4201
locked = db.execute("SELECT pg_try_advisory_lock(%s)", (JOB_LOCK_ID,)).fetchone()[0]
if not locked:
    raise SystemExit(0)
try:
    send_monthly_invoices()
finally:
    db.execute("SELECT pg_advisory_unlock(%s)", (JOB_LOCK_ID,))

How to check

Start the same job on two servers at once on staging. One run does the work and the other exits within 5 seconds, and no item is handled twice.

Sources

• Google SRE book, Distributed Periodic Scheduling with Cron
• Kubernetes, CronJob concurrency policy

Related rules

• Run web and worker processes apart in Scaling
• Spread scheduled work across the hour in Scaling
• Make writes safe to repeat in Data
• Alert when a scheduled job stops running in Operations