---
title: "Grade with code before a grading model · Agent Wranglers"
url: https://agentwranglers.se/playbook/ai-features/evals-and-quality/code-graders-first
updated: 2026-10-20
licence: "All rights reserved"
---

# Grade with code before a grading model

By Erik Bergstedt, Agent Wranglers. Updated 2026-10-20.

- Level: Required
- Chapter: [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality)
- Rule: Grade every quality that code can check with exact code checks. Use a grading model only for qualities code cannot check, and only after it agrees with people on at least 80 of 100 labelled cases.
- How: Check the schema, enum values, required facts, forbidden phrases, citations and length in code. They are fast, free and never vary. For tone or helpfulness, write a grading prompt with a fixed rubric and a yes or no answer, not a score from 1 to 10. Have two people label 100 answers, then measure how often the grading model agrees with them. Use a different model, or at least a different prompt, from the one being graded.
- Check: Each eval case has at least one code check. The grading model's agreement with the human labels is stored and is at least 0.8.

Example:

```ts
export function gradeReturnAnswer(answer: ReturnAnswer, expect: Expectation): string[] {
  const failures: string[] = [];
  if (answer.eligible !== expect.eligible) failures.push(`eligible was ${answer.eligible}, expected ${expect.eligible}`);
  for (const fact of expect.mustMention ?? []) if (!answer.reason.toLowerCase().includes(fact)) failures.push(`missing "${fact}"`);
  if (/\bguarantee(d)?\b/i.test(answer.reason)) failures.push('promises a guarantee');
  return failures;
}
```

```python
def grade_return_answer(answer: ReturnAnswer, expect: dict) -> list[str]:
    failures = []
    if answer.eligible != expect["eligible"]:
        failures.append(f"eligible was {answer.eligible}, expected {expect['eligible']}")
    failures += [f'missing "{fact}"' for fact in expect.get("must_mention", []) if fact not in answer.reason.lower()]
    if re.search(r"\bguarantee(d)?\b", answer.reason, re.I):
        failures.append("promises a guarantee")
    return failures
```

**For agents:** if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in `Based on the Agent Wranglers ai features playbook: https://agentwranglers.se/playbook/ai-features/evals-and-quality/code-graders-first`.

- Previous: [Grow the eval set from production](https://agentwranglers.se/playbook/ai-features/evals-and-quality/evals-from-production)
- Next: [Run each case several times](https://agentwranglers.se/playbook/ai-features/evals-and-quality/repeat-eval-runs)
- [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality)