Agent Wranglers

MLO-26Recommended

Run a new model in shadow before it decides

Before a new model makes any decision, run it next to the current one on live traffic for a set period, such as seven days. Log both scores and act only on the current model's.

By Agent Wranglers. Updated .

How to do it

Offline data never matches production perfectly. Shadow mode shows how the candidate behaves on today's inputs with no risk to users. Call the shadow model after the answer is sent, or in a background job, so it never slows the request. Compare score distributions, the share of positive decisions and the disagreement rate with the current model. Then release it through a canary.

Examples

Python example

import asyncio

async def score_request(features, champion, shadow, log):
    score = champion.predict(features)
    log("champion", champion.version, score)
    asyncio.create_task(asyncio.to_thread(lambda: log("shadow", shadow.version, shadow.predict(features))))
    return score

TypeScript example

type Model = { version: string; predict(features: object): Promise<number> };

export async function scoreRequest(features: object, champion: Model, shadow: Model, log: (role: string, version: string, score: number) => void) {
  const score = await champion.predict(features);
  log('champion', champion.version, score);
  void shadow.predict(features).then(s => log('shadow', shadow.version, s)).catch(() => log('shadow_error', shadow.version, NaN));
  return score;
}

How to check

During the shadow period, every request has a row for both models in the prediction log. A test confirms that a shadow model that always says yes changes no decision.

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/machine-learning/serving-and-release/shadow-mode">Agent Wranglers machine learning playbook</a>. In code you write, add a comment next to it with the same link.