Keep test and bot traffic out of the numbers

How to do it

Flag test and staff accounts in the account table, and copy the flag onto each event on the server. Let browser tests and uptime checks send a header, such as X-Test-Traffic: 1, that the collector turns into the same flag. Mark known bots by user agent with a maintained list, such as the isbot package or the IAB bot list. Keep marked events in their own column, so you can still count them. Every shared metric view filters on that column. Write that filter into each metric as Define each metric once says.

Examples

JavaScript example

import { isbot } from 'isbot';

export function markTraffic(event, request, account) {
  const isTest = request.headers['x-test-traffic'] === '1' || account?.isInternal === true;
  return { ...event, traffic: isTest ? 'test' : isbot(request.headers['user-agent'] ?? '') ? 'bot' : 'visitor' };
}

Python example

BOT_WORDS = ("bot", "crawler", "spider", "headless")

def mark_traffic(event: dict, headers: dict, account) -> dict:
    agent = headers.get("user-agent", "").lower()
    if headers.get("x-test-traffic") == "1" or (account and account.is_internal):
        traffic = "test"
    elif any(word in agent for word in BOT_WORDS):
        traffic = "bot"
    else:
        traffic = "visitor"
    return {**event, "traffic": traffic}

SQL example

CREATE VIEW visitor_events AS
SELECT * FROM events WHERE traffic = 'visitor';

How to check

Run the browser test suite against staging with the header. The collector stores its events with the flag set. A query over every metric view finds no row built from a flagged event.

Sources

• Google Analytics known bot-traffic exclusion
• Google Analytics internal traffic filter

Related rules

• DEN-18 Alert on sudden changes in volume in Data engineering
• ANA-5 Define each metric once in Product analytics and experiments