Give telemetry a budget and sample the bulk

How to do it

Telemetry bills grow with traffic and with each new label or log line, and one chatty release can double them. Measure ingest per service, such as gigabytes of logs and active metric series. Keep all warnings, errors and audit events. Sample routine success logs and traces, such as 1 in 10 for a busy route. Record the sample rate on each kept line, so counts can be scaled back. Sample traces after they end, as Trace requests across services says. Metric series grow with labels, so follow Keep metric labels few and bounded. Debug logging is switched on per service for a set time, never left on.

Examples

YAML example

# OpenTelemetry Collector: keep every error trace and 10% of the rest
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: some-of-the-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

JavaScript example

const SUCCESS_SAMPLE_RATE = 0.1;

export function logRequest(logger, entry) {
  if (entry.status < 400 && Math.random() >= SUCCESS_SAMPLE_RATE) return;
  logger.info({ ...entry, sample_rate: entry.status < 400 ? SUCCESS_SAMPLE_RATE : 1 });
}

Python example

import random

SUCCESS_SAMPLE_RATE = 0.1

def log_request(logger, entry: dict) -> None:
    is_success = entry["status"] < 400
    if is_success and random.random() >= SUCCESS_SAMPLE_RATE:
        return
    logger.info("request", extra={**entry, "sample_rate": SUCCESS_SAMPLE_RATE if is_success else 1})

How to check

A monthly report lists log, metric and trace volume per service against its budget. An alert fires when one passes 120%. A forced error on staging always appears in both logs and traces, whatever the sample rate.

Sources

• OpenTelemetry sampling

Related rules

• OPS-15 Trace requests across services in Operations and observability
• OPS-13 Keep metric labels few and bounded in Operations and observability