---
title: "Approve user feedback before it trains or · Agent Wranglers"
url: https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/approve-feedback-before-training
id: AIF-69
level: Recommended
applies: ai-features
updated: 2026-10-22
build: a2a6bdfed830
tokens: 1189
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Approve user feedback before it trains or teaches the model

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Logs, privacy and feedback](https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: AIF-69
- Level: Recommended
- Chapter: [Logs, privacy and feedback](https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback.md)
- Rule: User ratings and corrections reach a fine-tune, an eval answer or a prompt example only after a person approves each item. Cap the share of any set that one account supplies.
- Check: A test adds 200 thumbs-up ratings from one test account to a bad answer. The next training set holds none of them without approval, and at most 1% of its rows come from that account. Every row in each training set has an approver and an approval date.
- Applies when: The project uses AI models.
- How: Anyone with an account can send feedback. An attacker can rate bad answers up or send crafted corrections until the next tune learns them, without touching your servers. OWASP lists this under data and model poisoning. Keep incoming feedback in a holding table. A person approves each item before it is copied into a training set or an example. Cap what one account adds, such as 1% of a set, and set aside bursts of feedback from new accounts. Record which items went into each fine-tune. A bad batch can then be removed and the model rebuilt, as [Rebuild fine-tuned models from stored data](https://agentwranglers.se/playbook/ai-features/evals-and-quality/reproducible-fine-tunes.md) shows. Consent comes first, as [Train only on data people agreed to share](https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/consent-before-training.md) says.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### JavaScript example

```js
const MAX_SHARE_PER_ACCOUNT = 0.01;

export function trainingRows(feedback, setSize) {
  const perAccount = new Map();
  const limit = Math.floor(setSize * MAX_SHARE_PER_ACCOUNT);
  return feedback.filter(item => {
    if (!item.approvedBy) return false;
    const count = (perAccount.get(item.accountId) ?? 0) + 1;
    perAccount.set(item.accountId, count);
    return count <= limit;
  });
}
```

### Python example

```python
from collections import Counter

MAX_SHARE_PER_ACCOUNT = 0.01

def training_rows(feedback: list[Feedback], set_size: int) -> list[Feedback]:
    limit = int(set_size * MAX_SHARE_PER_ACCOUNT)
    per_account: Counter[str] = Counter()
    rows = []
    for item in feedback:
        if item.approved_by is None:
            continue
        per_account[item.account_id] += 1
        if per_account[item.account_id] <= limit:
            rows.append(item)
    return rows
```

## Sources

- [OWASP LLM05:2026 Data and Model Poisoning](https://github.com/GenAI-Security-Project/GenAI-LLM-Top10/blob/main/2026/final/LLM05_DataModelPoisoning.md)

## Related rules

- AIF-76 [Version the eval set and keep a held-out part](https://agentwranglers.se/playbook/ai-features/evals-and-quality/eval-set-versions.md) in AI features
- AIF-62 [Rebuild fine-tuned models from stored data and compare before switching](https://agentwranglers.se/playbook/ai-features/evals-and-quality/reproducible-fine-tunes.md) in AI features
- AIF-50 [Train only on data people agreed to share](https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/consent-before-training.md) in AI features
- MLO-43 [Limit what any one source can put into training data](https://agentwranglers.se/playbook/machine-learning/training-data/poisoning-checks.md) in Machine learning

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/approve-feedback-before-training">Agent Wranglers AI features playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Train only on data people agreed to share](https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/consent-before-training.md)
- Next: [Keep each customer's data in its own fine-tuned model](https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/tenant-fine-tunes.md)
- [Logs, privacy and feedback](https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback.md)