Alert on the age of the oldest waiting job

How to do it

A queue that grows slowly sends no errors, so emails, exports or reports arrive hours late before anyone notices. The length of a queue varies with traffic, but the age of its oldest item shows the delay users feel. Write the accepted delay as a freshness objective, as in set a target for each indicator. Export the age as a metric every minute, and alert as a ticket or a page by page only for what cannot wait. Items that keep failing go to a dead letter queue, so they never hold the age up.

Examples

JavaScript example

const oldest = await db.query(
  "SELECT EXTRACT(EPOCH FROM now() - min(created_at)) AS age FROM jobs WHERE status = 'waiting'"
);
queueAgeSeconds.set({ queue: 'emails' }, Number(oldest.rows[0].age ?? 0));

Python example

row = db.execute(
    "SELECT EXTRACT(EPOCH FROM now() - min(created_at)) FROM jobs WHERE status = 'waiting'"
).fetchone()
queue_age_seconds.labels(queue="emails").set(row[0] or 0)

How to check

Stop the workers of one queue on staging and add one job. The alert fires within the accepted delay plus 5 minutes, and clears within 5 minutes after the workers restart.

Sources

• Google SRE workbook, Data Processing Pipelines

Related rules

• Set a target for each indicator in Operations
• Page only for what cannot wait in Operations
• Set aside messages that keep failing in Scaling
• Test the service against every platform limit in Infrastructure
• Bound the backlog and push back on producers in Scaling
• Keep urgent jobs in their own queue in Scaling
• Scale on the resource that runs out in Scaling