evalsig reads the evaluation records of judges that do not always give the same answer — LLM judges, classifiers, rule sets — and answers, with intervals:
- how well the judge separates what humans call positive from what they call negative, and where it starts saying "positive";
- how often its answers split when the same input is judged again;
- whether a change really helped, or only moved the decision line or fell within noise.
It is an analysis-only command-line tool with a local web dashboard. It never calls an AI model. It treats human reference labels as noisy too: it measures how much they vary, states what each estimate targets, and compares systems and humans under the same rule.
When an AI agent improves a system in a loop, evalsig can act as a referee that keeps the holdout labels away from the agent: it seals them, counts every use, and decides pass or fail under conditions a human approved in advance.
Pre-release. The command-line interface, the record format and the output envelope are under construction and may change until 1.0.
From source, with Rust 1.90 or later (the minimum supported Rust version):
cargo install --locked evalsig-cliWith Homebrew, from the kechol/tap tap, which installs the release archive
for your platform:
brew install kechol/tap/evalsigPrebuilt binaries for macOS and Linux (aarch64 and x86_64) come with each GitHub Release, beside a SHA-256 checksum per archive and a shell installer. Every archive carries a GitHub artifact attestation; check where a download came from before trusting it as a referee:
gh attestation verify evalsig-cli-aarch64-apple-darwin.tar.xz --repo kechol/evalsigdocs/quickstart.md goes from an empty directory to a report and a comparison in under ten minutes. The specification is in docs/spec.md, the metric definitions in docs/metrics.md and every command and option in docs/cli.md.
| Question | Command |
|---|---|
| Can this ship? | gate (preregistered conditions on a sealed holdout), check (absolute thresholds with intervals) |
| Did the change help? | compare (paired intervals, minimum effect, transition table, pairwise win rate) |
| Better at telling cases apart, or just a moved threshold? | report (ROC and AUC, d′ and criterion c) |
| How good is it once human disagreement is included? Is it as close to humans as humans are to each other? | agree (reference modes, rater design, alternative annotator test) |
| How many items are enough? | power |
| Is production drifting? | trend, compare --unpaired |
- Calling AI models or running evaluations. Use Inspect, promptfoo or your own harness, then import the records.
- Prompt management, trace observation or a labelling interface (v1).
- A multi-user server or a cloud service.
Licensed under either of
- Apache License, Version 2.0 (LICENSE-APACHE)
- MIT license (LICENSE-MIT)
at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.