Non-functional tests Recommended
Fuzz every parser of untrusted input
Feed generated and random input to every parser of outside data, and turn every crash into a regular test.
How to do it
Use a coverage-guided fuzzer on file, header and message parsers, such as Go's built-in go test -fuzz, cargo-fuzz for Rust or Jazzer for Java. Use property-based tests, such as Hypothesis or fast-check, to state rules that must hold for any input. Run the fuzzer for a fixed time in CI, and keep the inputs that found bugs.
Example
// fast-check: decoding what we encoded always gives the original back
import fc from 'fast-check';
fc.assert(fc.property(fc.string(), text => decode(encode(text)) === text));
# Hypothesis
from hypothesis import given, strategies as st
@given(st.text())
def test_decoding_what_we_encoded_gives_it_back(text):
assert decode(encode(text)) == text
// Go native fuzzing
func FuzzRoundTrip(f *testing.F) {
f.Add("hello")
f.Fuzz(func(t *testing.T, text string) {
if decode(encode(text)) != text {
t.Fatalf("round trip changed %q", text)
}
})
}
How to check
The fuzz job runs in CI for its set time and finds no crash. Every past crash input is stored and passes as a unit test.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers testing playbook: https://agentwranglers.se/playbook/testing/non-functional-tests/fuzz-parsers.