Use one of these instead. They are real, finished, maintained software written by people who care about your use case:

Seriously. Go use hyperfine.

Fine. Here is the idea, so you can decide it's uninteresting and leave.

Premise: wall-clock time on a shared CI runner has a noise floor of roughly 10–20%, which is the same size as the regressions people want to catch. Every threshold-based CI benchmark therefore either cries wolf or sees nothing, and eventually gets muted.

Measured on a 32-core Linux box, six mise startup commands, comparing coefficient of variation across metrics:

Under 3.2× CPU oversubscription, wall-clock medians moved about 150%. Instruction counts moved 0.035%. So: gate CI on instruction counts, report wall clock without gating.

Re-measured through tak itself, benchmarking /bin/ls /usr in a container, quiet versus 32 spinning cores:

tests/counters.rs asserts that determinism on every CI run, and skips cleanly where valgrind is unavailable.

Corollary that cost me an afternoon to learn: syscall count and max RSS are not deterministic — they move with thread scheduling. Only instruction counts support a tight gate.

Second premise: the data belongs in the repository. tak stores results in refs/notes/tak, one JSON object per line, merged with git's cat_sort_uniq strategy so concurrent CI writers never conflict. No database, no account, no service.

This has a genuinely nice property. The notes tree is keyed by commit SHA as path names, so it does not reference the annotated commits at all — which means a single shallow fetch of one ref returns the entire history without cloning the repository:

$ git fetch --depth 1 origin '+refs/notes/tak:refs/notes/tak'
fetch of ONLY refs/notes/tak --depth 1:  36ms
refs present:                            1
project commit objects fetched:          0
notes cover:                             100 commits
on-disk size:                            124K

100 commits × 6 benchmarks packed to 124K. Ten thousand commits projects to roughly 6 MB.

Measuring and storing work. Nothing reads the data back yet.

• tak run — wall clock (min / p50 / mean / max), no shell
• instruction counts via cachegrind, degrading to timing-only where absent
• refs/notes/tak storage — run --record, push, history, init
• concurrent CI writers merge via cat_sort_uniq
• tak backfill --releases — benchmark published release binaries to bootstrap history
• tak compare <ref> — interleaved A/B against another revision
• bench.toml
• PR reporting
• change-point detection instead of thresholds

There is no usable cachegrind on Apple Silicon and none on Windows, so tak run there records timing only and says so. Run it in a container to get the deterministic metric on any host:

docker build -f docker/Dockerfile -t tak .
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD:/w" -w /w \
  tak run --bench startup -- ./mycli --help

The build context is the repository root, not docker/, because the build stage copies Cargo.toml and src/. And --user is not decoration: tak spawns the thing you are measuring, so the benchmark subject inherits the container's privileges over the mounted working tree.

The CI gate lives on the Linux job regardless, so this only matters locally.

• hyperfine — the UX everyone knows. Its own 2.0 roadmap asks for most of what's above.
• poop — hardware counters, no shell, first command as reference.
• git-appraise — one JSON object per line in git notes, merged with cat_sort_uniq. The storage design is lifted from here wholesale.
• git-perf — independently arrived at git-notes storage for performance data. Validates the approach; does not measure anything itself.
• Chronologer — benchmarking backwards across git history. Best idea in the field, sitting unused.
• Nyrkiö / Hunter — E-divisive change point detection, proven at DataStax, TigerBeetle, Confluent.

MIT