Agent Wranglers

TDD-17Recommended

Run the tests yourself and read 90 days of pipeline history

Run the full test suite on a clean checkout and record the pass rate and line and branch coverage of the critical paths. From the last 90 days of pipeline runs, record the failure rate, the flaky rate and the median duration.

By Agent Wranglers. Updated .

How to do it

A green badge on the README says little. The suite may skip half its tests, or the team may rerun failures until they pass. Run the suite yourself, on the build from Build the product from the company's own repository. Measure coverage per critical path, such as sign-in, payments and data export, not one number for the whole repository. Export the pipeline runs for the main branch. A flaky test is one that failed and then passed on the same commit. Compare what you find with Treat coverage as a floor and Quarantine flaky tests with an owner. Record the count of skipped and disabled tests too.

Examples

Python example

from statistics import median

def ci_health(runs: list[dict]) -> dict:
    by_commit: dict[str, set[str]] = {}
    for run in runs:
        by_commit.setdefault(run["commit"], set()).add(run["result"])
    flaky = sum(1 for results in by_commit.values() if {"failure", "success"} <= results)
    failed = sum(1 for run in runs if run["result"] == "failure")
    return {
        "failure_rate": failed / len(runs),
        "flaky_commit_rate": flaky / len(by_commit),
        "median_minutes": median(run["minutes"] for run in runs),
    }

JavaScript example

function ciHealth(runs) {
  const byCommit = new Map()
  for (const run of runs) byCommit.set(run.commit, (byCommit.get(run.commit) ?? new Set()).add(run.result))
  const flaky = [...byCommit.values()].filter(results => results.has('failure') && results.has('success')).length
  const minutes = runs.map(run => run.minutes).sort((a, b) => a - b)
  return {
    failureRate: runs.filter(run => run.result === 'failure').length / runs.length,
    flakyCommitRate: flaky / byCommit.size,
    medianMinutes: minutes[Math.floor(minutes.length / 2)],
  }
}

How to check

The report gives the test count, pass rate, skipped count, coverage per critical path, and the 90-day failure rate, flaky rate and median duration. Each number names the commit or the date range it came from.

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/due-diligence/code-and-ownership/tests-and-ci-health">Agent Wranglers technical due diligence playbook</a>. In code you write, add a comment next to it with the same link.