Split by time when the model predicts the future

How to do it

A random split puts next month's rows in the training set, so the model learns patterns it could not know yet. Pick the cutoff so the test period covers at least one full cycle the business has, such as a week or a season. Leave a gap between training and test as long as the label takes to arrive. For cross-validation, use expanding windows, such as scikit-learn's TimeSeriesSplit, instead of shuffled folds. Features must also be known at the cutoff, as Use only features known at the moment of prediction says.

Examples

Python example

from datetime import timedelta
import pandas as pd

def time_split(df: pd.DataFrame, cutoff: pd.Timestamp, label_delay: timedelta) -> tuple[pd.DataFrame, pd.DataFrame]:
    train = df[df["event_time"] < cutoff - label_delay]
    test = df[df["event_time"] >= cutoff]
    return train, test

TypeScript example

type Row = { eventTime: Date };

export function timeSplit<T extends Row>(rows: T[], cutoff: Date, labelDelayMs: number) {
  return {
    train: rows.filter(row => row.eventTime.getTime() < cutoff.getTime() - labelDelayMs),
    test: rows.filter(row => row.eventTime >= cutoff),
  };
}

How to check

A test reads the split of the latest training run. The newest training timestamp is earlier than the oldest test timestamp by at least the label delay.

Sources

• Google Rules of Machine Learning, rule 33
• scikit-learn cross-validation of time series data

Related rules

• MLO-12 Use only features known at the moment of prediction in Machine learning
• MLO-10 Keep each person or entity in one split in Machine learning