Test sequences of actions against a simple model

How to do it

A plain property test checks one call. Many bugs need a series of calls, such as add, remove, add again and then pay. List the actions a caller can take. Keep a model that does the same job in the simplest way, such as a dictionary of item counts. The tool runs random sequences against both and checks that they agree after every step. On a failure, it shrinks the sequence to the fewest steps that still fail. Hypothesis calls this a rule-based state machine, and fast-check calls it model-based testing. Copy each shrunk failure into a plain example test.

Examples

Python example

from hypothesis import strategies as st
from hypothesis.stateful import RuleBasedStateMachine, invariant, rule

SKUS = st.sampled_from(["mug", "shirt", "poster"])

class CartMachine(RuleBasedStateMachine):
    def __init__(self):
        super().__init__()
        self.cart = Cart()
        self.model = {}

    @rule(sku=SKUS, quantity=st.integers(min_value=1, max_value=5))
    def add(self, sku, quantity):
        self.cart.add(sku, quantity)
        self.model[sku] = self.model.get(sku, 0) + quantity

    @rule(sku=SKUS)
    def remove(self, sku):
        self.cart.remove(sku)
        self.model.pop(sku, None)

    @invariant()
    def counts_match_the_model(self):
        assert self.cart.counts() == self.model

TestCart = CartMachine.TestCase

JavaScript example

import fc from 'fast-check';
import { expect, test } from 'vitest';

class AddCommand {
  constructor(sku, quantity) { this.sku = sku; this.quantity = quantity; }
  check() { return true; }
  run(model, cart) {
    cart.add(this.sku, this.quantity);
    model[this.sku] = (model[this.sku] ?? 0) + this.quantity;
    expect(cart.counts()).toEqual(model);
  }
  toString() { return `add(${this.sku}, ${this.quantity})`; }
}

class RemoveCommand {
  constructor(sku) { this.sku = sku; }
  check() { return true; }
  run(model, cart) {
    cart.remove(this.sku);
    delete model[this.sku];
    expect(cart.counts()).toEqual(model);
  }
  toString() { return `remove(${this.sku})`; }
}

const anySku = fc.constantFrom('mug', 'shirt', 'poster');
const cartCommands = [
  fc.tuple(anySku, fc.integer({ min: 1, max: 5 })).map(([sku, quantity]) => new AddCommand(sku, quantity)),
  anySku.map(sku => new RemoveCommand(sku)),
];

test('the cart matches its model after any sequence of steps', () => {
  fc.assert(fc.property(fc.commands(cartCommands), steps => {
    fc.modelRun(() => ({ model: {}, real: new Cart() }), steps);
  }));
});

How to check

Each object on the risk list with three or more actions that change it has a stateful test. Break one action on purpose, such as removing an item twice. The test fails and prints a sequence of at most 10 steps.

Sources

• Hypothesis stateful tests
• fast-check model-based testing

Related rules

• Check rules that hold for any input with property tests in Testing
• Test what would hurt to break in Testing