Keep each person or entity in one split

How to do it

Ten orders from one customer look alike. If some train and some test, the model scores well by recognising the customer, and fails on new ones. Assign the split from a hash of the entity ID, so the same entity lands in the same split in every snapshot. In cross-validation, use group folds, such as scikit-learn's GroupKFold. Combine this with Split by time when the model predicts the future when both apply.

Examples

Python example

import hashlib

TEST_SHARE_PERCENT = 20

def split_of(entity_id: str) -> str:
    bucket = int(hashlib.sha256(entity_id.encode()).hexdigest(), 16) % 100
    return "test" if bucket < TEST_SHARE_PERCENT else "train"

Go example

const testSharePercent = 20

func SplitOf(entityID string) string {
	sum := sha256.Sum256([]byte(entityID))
	bucket := new(big.Int).Mod(new(big.Int).SetBytes(sum[:]), big.NewInt(100)).Int64()
	if bucket < testSharePercent {
		return "test"
	}
	return "train"
}

How to check

A query joins the training and test sets of the latest run on the entity key. It returns no rows.

Sources

• scikit-learn cross-validation for grouped data

Related rules

• MLO-9 Split by time when the model predicts the future in Machine learning
• MLO-13 Remove duplicates across splits in Machine learning