Validate every event before you store it

How to do it

Compile the tracking plan into JSON Schema 2020-12 validators, such as Ajv's Ajv2020 class or Python's jsonschema. Reject unknown events, unknown properties and wrong types. Write rejects to a separate table with the error and the app version, as with a dead letter queue. Alert when rejects pass 1 percent of events in an hour, because a release has broken tracking. It is the parse at the boundary rule applied to events.

Examples

JavaScript example

import Ajv2020 from 'ajv/dist/2020.js';

const ajv = new Ajv2020({ allErrors: true });
const validators = Object.fromEntries(Object.entries(EVENT_SCHEMAS).map(([name, schema]) => [name, ajv.compile(schema)]));

export async function collect(event) {
  const validate = validators[event.name];
  if (validate?.(event.properties)) return db.events.insert(event);
  return db.eventRejects.insert({ event, error: validate ? ajv.errorsText(validate.errors) : 'unknown event' });
}

Python example

from jsonschema import Draft202012Validator

VALIDATORS = {name: Draft202012Validator(schema) for name, schema in EVENT_SCHEMAS.items()}

def collect(event: dict):
    validator = VALIDATORS.get(event["name"])
    errors = [e.message for e in validator.iter_errors(event["properties"])] if validator else ["unknown event"]
    if errors:
        return db.event_rejects.insert(event=event, error="; ".join(errors))
    return db.events.insert(event)

How to check

Send one valid and one invalid event to the staging collector. The valid one is in the events table, and the invalid one is in the rejects table with its error.

Sources

• JSON Schema 2020-12

Related rules

• EVT-23 Set aside messages that keep failing in Integration and events
• ARC-53 Parse input once at the boundary in Clean code and architecture