Alert when a consumer falls behind

How to do it

A stuck consumer raises no error. The broker keeps taking messages, and the backlog grows until a customer notices a missing email. Lag in messages is the newest offset minus the committed offset. Lag in time says more, because a busy partition can hold 10,000 messages that are 2 seconds old. Export both per group and per partition, for example with a lag exporter or the broker's own metrics. Alert on time, from the delay each consumer promises, such as 5 minutes for order emails. A lag that grows on one partition only points at a hot key, as Choose the partition key for the order you need explains. Job queues follow Alert on the age of the oldest waiting job.

Examples

YAML example

# Prometheus alert on consumer lag in seconds, from a lag exporter
- alert: OrderEmailsConsumerBehind
  expr: max by (group, partition) (kafka_consumergroup_lag_seconds{group="order-emails"}) > 300
  for: 2m
  labels: { severity: page }
  annotations: { summary: "order-emails is {{ $value }}s behind on partition {{ $labels.partition }}" }

Python example

def lag_seconds(newest_timestamp_ms: int, committed_timestamp_ms: int) -> float:
    return max(0, newest_timestamp_ms - committed_timestamp_ms) / 1000

MAX_LAG_SECONDS = 300
if lag_seconds(newest, committed) > MAX_LAG_SECONDS:
    alert("order-emails consumer is behind", partition=partition)

JavaScript example

const MAX_LAG_SECONDS = 300;

export function checkLag({ group, partition, newestTimestampMs, committedTimestampMs }) {
  const lagSeconds = Math.max(0, newestTimestampMs - committedTimestampMs) / 1000;
  if (lagSeconds > MAX_LAG_SECONDS) alert(`${group} is ${lagSeconds}s behind on partition ${partition}`);
}

How to check

Each consumer group has a lag alert with a written time limit. Stop one consumer on staging while the producer runs. The alert fires before the limit, and it names the group and the partition.

Sources

• Apache Kafka design, the consumer position

Related rules

• EVT-9 Choose the partition key for the order you need in Integration and events
• OPS-67 Alert on the age of the oldest waiting job in Operations and observability