flakescopeOpen source

A flaky test fails sometimes and passes sometimes, and the useful question is what it depends on. flakescope answers that for Go packages. It varies configurations, not goroutine interleavings: Go has no seedable goroutine scheduler outside testing/synctest, so no tool can replay a particular ordering. What it can vary is test order, available processors, and the race detector, and it reports which of those a failure tracks.

A configuration matrix laid out over cells, one per shuffle arm and GOMAXPROCS candidate, with runs dealt across cells in rotation so every arm collects enough observations to state a failure rate.

A classifier that separates flaky tests from always-failing ones and labels each flaky test order-dependent, load-dependent, both, or undetermined, printing the evidence rates next to the label.

Failure clustering by normalized signature with a smallest observed reproducing configuration per cluster, plus a machine-readable --json report on a versioned schema.

flakescope generates the configuration matrix, runs go test -json for each configuration across parallel workers, and folds the event streams into per-test pass and fail counts broken out by axis. Classification, clustering, and minimal-configuration selection all run over that aggregate. Its own test suite replays recorded go test -json streams from testdata rather than invoking go test, so the tool that detects flaky tests does not have flaky tests of its own.

Dependence is decided by failure rate, not presence. Until v0.3.0 the order rule only asked whether every failure happened under shuffle; with most configurations shuffled, that was true by chance for almost anything that failed at all, so nearly every flaky test came back order-dependent. The fix compares rates between arms and requires at least four observations per arm, a 2x ratio, and a pooled two-proportion z of 2.

Clustering prefers splitting over merging. A split shows up as two near-identical clusters you can read; a merge hides a second bug behind the first. So only run-varying noise — goroutine IDs, addresses, temp paths, durations — is normalized, and integers in messages are left alone.

The race detector runs on one configuration in seven. Race builds dominate wall-clock time and answer a yes-or-no question that a sample handles as well as a census, which cuts matrix cost from 5.5x a race-free run to 2.3x. Seven is odd so raced runs do not all land in the same shuffle arm.

A 1,000-configuration run against the fixture package produced 2,174 individual failures across three tests and reported them as four clusters, including a load-dependent test split into its two distinct failure modes at GOMAXPROCS 4 and 2.

The report states its own resolution limit rather than overclaiming: at the default 20 runs it can only separate failure rates above roughly 38 percent on the shuffle axis and 50 percent on GOMAXPROCS, and when no axis separates it says undetermined and prints the rate that run could have resolved, so the next --runs is a number rather than a hunch.