Compare old and new answers before switching a route

How to do it

The proxy or a small script replays read requests against the new system and logs any difference, while visitors still get the old answer. Never replay writes, since they would run twice. Mask fields that always differ, such as request IDs and timestamps, as Mask volatile parts narrowly says. Each difference is a bug in the new code or a pinned old bug, as Keep known bugs pinned until their own fix says. After the switch, follow Compare the live system before and after each deploy.

Examples

JavaScript example

const VOLATILE = new Set(['requestId', 'generatedAt']);
const stable = body => JSON.stringify(body, (key, value) => (VOLATILE.has(key) ? undefined : value));

export async function compareRoute(path, headers) {
  const [legacy, next] = await Promise.all([fetch(LEGACY + path, { headers }), fetch(NEW + path, { headers })]);
  const [a, b] = await Promise.all([legacy.json(), next.json()]);
  if (legacy.status !== next.status || stable(a) !== stable(b)) log.warn('route differs', { path });
}

Python example

VOLATILE = {"requestId", "generatedAt"}

def stable(body):
    if isinstance(body, dict):
        return {key: stable(value) for key, value in body.items() if key not in VOLATILE}
    if isinstance(body, list):
        return [stable(item) for item in body]
    return body

def compare_route(path, headers):
    legacy, new = httpx.get(LEGACY + path, headers=headers), httpx.get(NEW + path, headers=headers)
    if legacy.status_code != new.status_code or stable(legacy.json()) != stable(new.json()):
        log.warning("route differs", extra={"path": path})

How to check

For each moved route, the comparison log covers at least 1,000 requests or 7 days of traffic before the switch. Every difference in it has a ticket or a written reason.

Related rules

• TST-44 Mask volatile parts narrowly in Testing
• TKO-9 Keep known bugs pinned until their own fix in Codebase takeover
• TST-45 Compare the live system before and after each deploy in Testing