Go Benchmark Comparison
Two benchmark numbers cannot tell you whether a change is real. Paste two runs of `go test -bench -count=10` and get the same statistical comparison benchstat makes — median change, p-value, and a plain refusal when the data cannot answer.
This is not the benchstat binary. It implements the same statistical test — Mann-Whitney U, two-sided, at the same conventional 5% level — over the same input format, and it reports the same shape of answer. It is not golang.org/x/perf/cmd/benchstat and it is not derived from that code. Where the two disagree, believe the one with the Go team's name on it.
Nothing you paste is transmitted. There is no endpoint behind this page: the Go that does the statistics is compiled to WebAssembly and runs in this tab, so your benchmark output never leaves the browser.
Start from one of these
Why a single number proves nothing
Run the same benchmark twice on the same binary and you get different numbers. Run it on a laptop that decided to throttle and you get very different numbers. So “1250 ns/op became 1210 ns/op, a 3% win” is not a result — it is two samples from a noisy process, and 3% is well inside what the noise does on its own.
The question that can be answered is a different one: given these samples, how surprising would this separation be if both runs came from the same distribution? That is what the Mann-Whitney U test computes, and the answer is the p-value in the table above.
Why this test and not a t-test
Mann-Whitney U is a rank test: it looks at how the two samples interleave when sorted, not at their arithmetic. That matters because benchmark timings are not normally distributed. Noise in a benchmark is almost entirely one-sided — an interrupted run is slow, never fast — so the distribution has a long right tail and a t-test's assumptions do not hold.
The same reasoning is why the summary column is the median rather than the mean. One run that got descheduled can move a mean by ten percent while telling you nothing about the code.
Exact, where it can be
With no tied values and small samples, the p-value here is computed from the exact null distribution — by counting how many of the possible interleavings are at least as extreme — rather than from a normal approximation. Ten runs of each is a lot for a person and very little for an approximation, so this matters at exactly the sample sizes people actually have.
Ties force the approximation, with the standard tie correction, and the table says which was used. Identical values are common in practice: allocation counts almost never move.
One run of each
Cannot be tested, and the tool says so instead of printing a number. With one sample on each side there is no separation to be surprised by. That is not a limitation to work around — it is the honest answer to the question, and the fix is -count=10.
“No significant difference” is not “no difference”
A result that fails to reach p < 0.05 means this data does not distinguish the two runs. A real improvement smaller than your noise looks exactly the same. If you need to detect a 2% change, you need enough runs to see through 2% of noise — and the way to find out how many is to run more and watch the p-value move.
The other tools
- Go Time Layout Converter — strftime to Go, and back — Convert a C strftime format string to a Go reference layout and back, directive by directive, with every unmappable directive named rather than guessed.
- Go Method Sets — why *T implements the interface and T does not — Paste a type and an interface and get both method sets computed by go/types, with the compiler's own error and the rule it follows from.
- Go Struct Field Alignment — offsets, padding, and what reordering saves — Paste a struct and get every field's offset, the padding between them, the total size, and a reordered version with the saving — for the architecture you choose.
- Go Slice Growth — what append actually does to the capacity — Watch a slice's capacity grow append by append: the formula's answer, the allocator's rounding, the size class it lands in, and the bytes left behind.
- Go Goroutine Dump Analyzer — read a SIGQUIT or panic stack dump — Paste a goroutine dump and get it grouped by state and by stack, with wait durations and the stacks held by far more goroutines than they should be.
- GOGC and GOMEMLIMIT Calculator — which one is actually deciding your heap — Compute both documented target-heap rules, see which one binds, and find out when GOGC has stopped having any effect at all.
- Go Minimal Version Selection — why that version, and who asked for it — Paste a go mod graph and see minimal version selection worked out: every version required, who required it, and the path from your main module to the requirement that won.