---
title: "Mark automated decisions and review a · Agent Wranglers"
url: https://agentwranglers.se/playbook/trust-and-safety/running-trust-and-safety/automated-decision-sampling
id: TNS-21
level: Recommended
updated: 2026-10-27
build: a2a6bdfed830
tokens: 1247
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Mark automated decisions and review a sample of them

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Running trust and safety](https://agentwranglers.se/playbook/trust-and-safety/running-trust-and-safety.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: TNS-21
- Level: Recommended
- Chapter: [Running trust and safety](https://agentwranglers.se/playbook/trust-and-safety/running-trust-and-safety.md)
- Rule: Record on every decision whether automated means found the item and whether they decided it. Each week, give a person a random sample of 1% of automated removals and 1% of items the filters passed, with the filter's verdict hidden. Take at least 20 of each, or all of them when there are fewer, and at most 100.
- Check: Every decision row from the last 30 days holds both automation fields. A unit test of the weekly job gives it 500 removals and 50,000 passed items and gets 20 and 100 review tasks. With 8 removals it takes all 8. No task shows the filter's verdict. A dashboard shows the error rates per filter for the last four weeks.
- How: The DSA Transparency Database asks two fields of each statement of reasons. `automated_detection` is yes or no, and `automated_decision` is fully, partially or not automated. Store both on the decision row, so statements and reports come from the data. Appeals in [Have a second person decide each appeal](https://agentwranglers.se/playbook/trust-and-safety/decisions-and-appeals/appeal-review.md) show only the errors that users contest. A blind sample also finds wrong removals nobody appealed and harmful items the filter missed. Compute the share of wrong removals and of missed items per filter and reason. Send a filter's decisions to a person while its wrong removals pass 5%, as [Let a person decide what matters](https://agentwranglers.se/playbook/ai-features/guardrails-and-human-review/human-decides.md) says. Put the rates in the [yearly moderation report](https://agentwranglers.se/playbook/compliance/online-platforms/moderation-report.md).

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### JavaScript example

```js
const SAMPLE_SHARE = 0.01;
const MIN_SAMPLE = 20;
const MAX_SAMPLE = 100;

export const sampleSize = total => Math.min(total, MAX_SAMPLE, Math.max(MIN_SAMPLE, Math.ceil(total * SAMPLE_SHARE)));

export async function weeklyReviewSample() {
  const removed = decisions.automated({ sinceDays: 7 });
  const passed = items.passedByFilters({ sinceDays: 7 });
  const picked = [
    ...await removed.sample(sampleSize(await removed.count())),
    ...await passed.sample(sampleSize(await passed.count())),
  ];
  for (const item of picked) await reviewTasks.create({ itemId: item.itemId, showVerdict: false });
}
```

### Python example

```python
SAMPLE_SHARE = 0.01
MIN_SAMPLE = 20
MAX_SAMPLE = 100

def sample_size(total: int) -> int:
    return min(total, MAX_SAMPLE, max(MIN_SAMPLE, math.ceil(total * SAMPLE_SHARE)))

def weekly_review_sample():
    removed = decisions.automated(since_days=7)
    passed = items.passed_by_filters(since_days=7)
    picked = [*removed.sample(sample_size(removed.count())), *passed.sample(sample_size(passed.count()))]
    for item in picked:
        review_tasks.create(item_id=item.item_id, show_verdict=False)
```

## Sources

- [DSA Transparency Database API documentation](https://transparency.dsa.ec.europa.eu/page/api-documentation)

## Related rules

- TNS-10 [Have a second person decide each appeal](https://agentwranglers.se/playbook/trust-and-safety/decisions-and-appeals/appeal-review.md) in Trust and safety
- AIF-36 [Let a person decide what matters](https://agentwranglers.se/playbook/ai-features/guardrails-and-human-review/human-decides.md) in AI features
- CMP-65 [Publish a yearly moderation report](https://agentwranglers.se/playbook/compliance/online-platforms/moderation-report.md) in Compliance and governance

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/trust-and-safety/running-trust-and-safety/automated-decision-sampling">Agent Wranglers trust and safety playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Watch abuse rates and alert on jumps](https://agentwranglers.se/playbook/trust-and-safety/running-trust-and-safety/abuse-rate-alerts.md)
- Next: [Label content made by AI from its content credentials](https://agentwranglers.se/playbook/trust-and-safety/running-trust-and-safety/ai-content-labels.md)
- [Running trust and safety](https://agentwranglers.se/playbook/trust-and-safety/running-trust-and-safety.md)