---
title: "Record the compute cost of every training · Agent Wranglers"
url: https://agentwranglers.se/playbook/machine-learning/training-runs-and-the-registry/training-run-cost
id: MLO-45
level: Recommended
applies: ai-features
updated: 2026-10-27
build: a2a6bdfed830
tokens: 831
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Record the compute cost of every training run

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Training runs and the registry](https://agentwranglers.se/playbook/machine-learning/training-runs-and-the-registry.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: MLO-45
- Level: Recommended
- Chapter: [Training runs and the registry](https://agentwranglers.se/playbook/machine-learning/training-runs-and-the-registry.md)
- Rule: Record the machine hours and the cost of every training and tuning run in its run record. Give each model a budget per run, and stop a run that passes it.
- Check: Every run record in the last 30 days has machine hours and a cost. A test run with a budget below one checkpoint's cost stops at the first checkpoint with a budget status.
- Applies when: The project uses AI models.
- How: Tuning sweeps and retraining on a schedule can cost more than serving the model, and nobody sees it until the bill. Read the hours from the job scheduler and the price from your cloud price list, and write both to the [run record](https://agentwranglers.se/playbook/machine-learning/training-runs-and-the-registry/training-run-record.md). Put the budget in the training config. Check spend at each checkpoint and stop with a clear status when it passes. Report cost per model per month next to its business value, so a retraining schedule can be judged on cost too.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### Python example

```python
def on_checkpoint(run: TrainingRun) -> None:
    cost = run.gpu_hours() * PRICE_PER_GPU_HOUR[run.machine_type]
    mlflow.log_metric("cost_usd", cost)
    if cost > run.config.budget_usd:
        run.stop(status="budget_limit")
```

### TypeScript example

```ts
export function onCheckpoint(run: TrainingRun) {
  const costUsd = run.gpuHours() * PRICE_PER_GPU_HOUR[run.machineType];
  tracker.logMetric(run.id, 'cost_usd', costUsd);
  if (costUsd > run.config.budgetUsd) run.stop('budget_limit');
}
```

## Sources

- [FinOps Foundation, FinOps for AI overview](https://www.finops.org/wg/finops-for-ai-overview/)
- [MLflow tracking](https://mlflow.org/docs/latest/ml/tracking/)

## Related rules

- MLO-20 [Record how every model was trained](https://agentwranglers.se/playbook/machine-learning/training-runs-and-the-registry/training-run-record.md) in Machine learning

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/machine-learning/training-runs-and-the-registry/training-run-cost">Agent Wranglers machine learning playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Store models in formats that cannot run code](https://agentwranglers.se/playbook/machine-learning/training-runs-and-the-registry/safe-model-formats.md)
- [Training runs and the registry](https://agentwranglers.se/playbook/machine-learning/training-runs-and-the-registry.md)