---
title: "Give telemetry a budget and sample the · Agent Wranglers"
url: https://agentwranglers.se/playbook/operations/metrics-and-traces/telemetry-budget
id: OPS-81
level: Recommended
updated: 2026-10-27
build: a2a6bdfed830
tokens: 1027
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Give telemetry a budget and sample the bulk

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Metrics and traces](https://agentwranglers.se/playbook/operations/metrics-and-traces.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: OPS-81
- Level: Recommended
- Chapter: [Metrics and traces](https://agentwranglers.se/playbook/operations/metrics-and-traces.md)
- Rule: Set a monthly volume budget for each service's logs, metric series and traces. Keep every error and a fixed share of successful requests, and alert when a service passes 120% of its budget.
- Check: A monthly report lists log, metric and trace volume per service against its budget. An alert fires when one passes 120%. A forced error on staging always appears in both logs and traces, whatever the sample rate.
- How: Telemetry bills grow with traffic and with each new label or log line, and one chatty release can double them. Measure ingest per service, such as gigabytes of logs and active metric series. Keep all warnings, errors and audit events. Sample routine success logs and traces, such as 1 in 10 for a busy route. Record the sample rate on each kept line, so counts can be scaled back. Sample traces after they end, as [Trace requests across services](https://agentwranglers.se/playbook/operations/metrics-and-traces/distributed-tracing.md) says. Metric series grow with labels, so follow [Keep metric labels few and bounded](https://agentwranglers.se/playbook/operations/metrics-and-traces/bounded-metric-labels.md). Debug logging is switched on per service for a set time, never left on.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### YAML example

```yaml
# OpenTelemetry Collector: keep every error trace and 10% of the rest
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: some-of-the-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }
```

### JavaScript example

```js
const SUCCESS_SAMPLE_RATE = 0.1;

export function logRequest(logger, entry) {
  if (entry.status < 400 && Math.random() >= SUCCESS_SAMPLE_RATE) return;
  logger.info({ ...entry, sample_rate: entry.status < 400 ? SUCCESS_SAMPLE_RATE : 1 });
}
```

### Python example

```python
import random

SUCCESS_SAMPLE_RATE = 0.1

def log_request(logger, entry: dict) -> None:
    is_success = entry["status"] < 400
    if is_success and random.random() >= SUCCESS_SAMPLE_RATE:
        return
    logger.info("request", extra={**entry, "sample_rate": SUCCESS_SAMPLE_RATE if is_success else 1})
```

## Sources

- [OpenTelemetry sampling](https://opentelemetry.io/docs/concepts/sampling/)

## Related rules

- OPS-15 [Trace requests across services](https://agentwranglers.se/playbook/operations/metrics-and-traces/distributed-tracing.md) in Operations and observability
- OPS-13 [Keep metric labels few and bounded](https://agentwranglers.se/playbook/operations/metrics-and-traces/bounded-metric-labels.md) in Operations and observability

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/operations/metrics-and-traces/telemetry-budget">Agent Wranglers operations and observability playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Record deploys on the graphs](https://agentwranglers.se/playbook/operations/metrics-and-traces/deploy-markers.md)
- Next: [Profile production all the time at low cost](https://agentwranglers.se/playbook/operations/metrics-and-traces/continuous-profiling.md)
- [Metrics and traces](https://agentwranglers.se/playbook/operations/metrics-and-traces.md)