Keep metric labels few and bounded

How to do it

Every new label value creates a new time series, which costs memory and money and can stop the metrics store. Label by the route template, such as /orders/:id, never the raw path. Send requests that match no route to one value, such as unmatched. Put per-user detail in logs and traces instead. Agree a limit per metric, such as 1,000 series.

Example

import client from 'prom-client';

const duration = new client.Histogram({
  name: 'http_server_request_duration_seconds',
  help: 'Time to answer a request',
  labelNames: ['http_route', 'http_request_method', 'http_response_status_code'],
  buckets: [0.05, 0.1, 0.3, 1, 3],
});

app.use((req, res, next) => {
  const stopTimer = duration.startTimer();
  res.on('finish', () => stopTimer({
    http_route: req.route ? req.baseUrl + req.route.path : 'unmatched',
    http_request_method: req.method,
    http_response_status_code: res.statusCode,
  }));
  next();
});

import time
from flask import g, request
from prometheus_client import Histogram

DURATION = Histogram(
    "http_server_request_duration_seconds", "Time to answer a request",
    ["http_route", "http_request_method", "http_response_status_code"],
    buckets=[0.05, 0.1, 0.3, 1, 3],
)

@app.before_request
def start_timer():
    g.started = time.perf_counter()

@app.after_request
def record_duration(response):
    route = request.url_rule.rule if request.url_rule else "unmatched"
    DURATION.labels(route, request.method, response.status_code).observe(time.perf_counter() - g.started)
    return response

How to check

A series count per metric, such as topk(10, count by (__name__)({__name__=~".+"})) in Prometheus, shows none above the limit. No label is named user_id, email, path or request_id.