Test speech features with recordings from many speakers

How to do it

The word error rate is the substituted, dropped and added words divided by the words in the correct transcript. Grow the set from production clips people agreed to share, as Grow the eval set from production describes. Include the words your feature lives on, such as product names and street names. Compare the groups the way Test that answers do not change with who asks compares answers.

Examples

TypeScript example

export function wordErrorRate(reference: string, heard: string): number {
  const ref = reference.toLowerCase().split(/\s+/).filter(Boolean);
  const hyp = heard.toLowerCase().split(/\s+/).filter(Boolean);
  let previous = Array.from({ length: hyp.length + 1 }, (_, index) => index);
  for (let i = 1; i <= ref.length; i++) {
    const current = [i];
    for (let j = 1; j <= hyp.length; j++) {
      current[j] = Math.min(previous[j] + 1, current[j - 1] + 1, previous[j - 1] + (ref[i - 1] === hyp[j - 1] ? 0 : 1));
    }
    previous = current;
  }
  return previous[hyp.length] / ref.length;
}

Python example

def word_error_rate(reference: str, heard: str) -> float:
    ref, hyp = reference.lower().split(), heard.lower().split()
    previous = list(range(len(hyp) + 1))
    for i in range(1, len(ref) + 1):
        current = [i]
        for j in range(1, len(hyp) + 1):
            current.append(min(previous[j] + 1, current[j - 1] + 1,
                               previous[j - 1] + (ref[i - 1] != hyp[j - 1])))
        previous = current
    return previous[-1] / len(ref)

How to check

The eval report lists the word error rate per group. No group's rate is more than 1.5 times the best group's, or a decision note explains the gap and the plan.

Sources

• NIST AI 100-1, AI Risk Management Framework

Related rules

• AIF-26 Grow the eval set from production in AI features
• AIF-32 Test that answers do not change with who asks in AI features