marmot bench
Run benchmarks and compare them with go or a saved run.
Build a package's benchmarks with marmot, run them several times, and report the median and spread of each. -vs go also builds them with go test and runs both alternately, then reports the change and its significance (Mann-Whitney U, benchstat style). -save keeps the raw results; -compare FILE compares with them.
marmot bench [package] [flags]Examples
marmot bench -vs go ./...
marmot bench -bench Fib -count 10 .
marmot bench -save old.txt . && marmot bench -compare old.txt .marmot bench -vs go ./pkg builds the package's benchmarks with marmot and with go test, runs
them alternately (-count, default 6), and prints the median and spread of each, the change and
whether it is significant (Mann-Whitney U, as benchstat does).
Benchmarks are built at -O 2 unless you give a level or -release. A benchmark under 1 ns
per iteration gets a note: the optimizer probably removed work whose result was unused.
-cpuprofile FILE profiles the last run; read it with marmot prof.
marmot bench -vs go -cpuprofile cpu.out ./pkg
marmot prof -tree cpu.outFlags
Flags take one dash or two: -release and --release are the same.
| Flag | Meaning |
|---|---|
-bench string | regexp of the benchmarks to run (default ".") |
-benchtime string | time or iterations per benchmark run, as -test.benchtime |
-color string | color in human output: auto, always or never (NO_COLOR turns auto off) (default "auto") |
-compare string | compare with results saved by -save (or any go test -bench output) |
-count string | runs of each benchmark (more runs, smaller p-values) (default "6") |
-cpuprofile string | write a CPU profile of the last run to this file (with -vs go, the go build's to FILE.go); read it with marmot prof |
-json | machine readable output: JSON Lines on stderr (also $MARMOT_OUTPUT=json, and the default under coding agents) |
-save string | write the raw results to this file |
-timeout string | time limit of each benchmark binary run (default "30m") |
-vs string | go: also run the benchmarks built by go test and compare |
Full help
Build a package's benchmarks with marmot, run them several times, and report the median and spread of each. -vs go also builds them with go test and runs both alternately, then reports the change and its significance (Mann-Whitney U, benchstat style). -save keeps the raw results; -compare FILE compares with them.
Usage:
marmot bench [package] [flags]
Examples:
marmot bench -vs go ./...
marmot bench -bench Fib -count 10 .
marmot bench -save old.txt . && marmot bench -compare old.txt .
Flags:
--bench string regexp of the benchmarks to run (default ".")
--benchtime string time or iterations per benchmark run, as -test.benchtime
--color string color in human output: auto, always or never (NO_COLOR turns auto off) (default "auto")
--compare string compare with results saved by -save (or any go test -bench output)
--count string runs of each benchmark (more runs, smaller p-values) (default "6")
--cpuprofile string write a CPU profile of the last run to this file (with -vs go, the go build's to FILE.go); read it with marmot prof
-h, --help help for bench
--json machine readable output: JSON Lines on stderr (also $MARMOT_OUTPUT=json, and the default under coding agents)
--save string write the raw results to this file
--timeout string time limit of each benchmark binary run (default "30m")
--vs string go: also run the benchmarks built by go test and compare