Loghub SRE Harbor benchmark

A Harbor-compatible benchmark for SRE log-investigation skills. 180 scored tasks built from the Loghub corpus (logpai/loghub), across 6 skill types × 5 datasets: anomaly localization (v1, 60), false-positive triage, temporal-sequence reconstruction, cross-component correlation, severity classification (v2, 100), and outcome-oriented remediation (v3, 20). A 7th type, log-template extraction, ships as a 20-task unscored tooling track (tracks/tooling/) — see why.

Tasks carry opaque ids (lh-<hash>) so the slug leaks no dataset or root-cause hint; the human-readable mapping is in docs/task-id-map.json.

Each task ships a Docker environment with partitioned log files and asks the agent to investigate and emit a structured JSON answer (/app/answer.json). The verifier grades it with gate-aware scoring: format/integrity checks are gates (any failure → reward 0, passing earns no credit), and the fractional reward (passed / non_skipped over the substantive tests) is written to /logs/verifier/reward.txt. A schema-compliant no-investigation answer scores 0.000. Full layout and grading: docs/structure-and-grading.md.

Committed tasks set [environment].allow_internet = false; verifier dependencies are baked into the task image, and agent/runtime dependencies must come from the local harness setup rather than task-time installs.

Status: ready for community testing. The committed set passes every gate — adapter/exporter/repo-invariant tests, make static (12 checks × 180 tasks), and make oracle-nop (oracle 180/180, nop 0/180).

v1.0 re-baseline in progress. The rows below were measured under the pre-v1.0 reward (passed / non_skipped, zero-effort floor 0.62). v1.0 switches to gate-aware scoring (zero-effort floor 0.000 — schema compliance earns nothing) and diversified remediation actions, so these numbers are not comparable to v1.0 runs. The v1.0 table will report fully-solved rate as the headline metric, mean reward with 95% CIs over ≥3 runs, per-row cost, and a published zero-effort floor row. Leaderboard protocol: docs/scoring.md.

180-task runs (May 2026). "Errors" are Harbor trial exceptions counted as zero reward. Committed run artifacts are linked. Harness: harbor==0.13.1.

Reproduce a baseline:

export DEEPSEEK_API_KEY=<your key>
harbor run -p tasks/ --agent mini-swe-agent -m deepseek/deepseek-v4-flash \
  --agent-timeout-multiplier 3.0 -o baselines

Scored set (180):

Plus an unscored tooling track: log-template extraction (20), in tracks/tooling/ — kept out of the leaderboard because its ground truth comes from the published Loghub-2k templates (contamination) and a deterministic templater solves it (mechanizable). Distribution and rationale: docs/dataset-adapters.md, docs/data-provenance.md.

git clone https://github.com/Cerebral-Systems/loghub-benchmark
cd loghub-benchmark
uv venv .venv-tools && . .venv-tools/bin/activate
pip install 'harbor==0.13.1' pytest pytest-json-ctrf   # pinned harness + verifier deps
make unit            # adapter + exporter + invariant tests (instant)

Running the benchmark needs Docker (and the corpus on disk only to regenerate tasks — see docs/data-setup.md).

harbor run -p tasks/lh-f53d4948 --agent oracle    # reward=1
harbor run -p tasks/lh-f53d4948 --agent nop       # reward=0
make validate-all                                           # all CI gates locally

To re-render the per-task verifier/test files in place from the export_to_harbor templates (no corpus needed; CI asserts they never drift):

python -m tools.case_builder.rebuild_tests          # write in place
python -m tools.case_builder.rebuild_tests --check  # drift gate

Task anatomy, answer schemas, the verifier/reward model, evidence-validation modes, and the determinism/canary/anti-leak gates are documented in docs/structure-and-grading.md (exhaustive per-assertion list in docs/scoring.md).

tasks/lh-<hash>/        # opaque id — the slug encodes nothing about the answer
├── instruction.md     # what the agent sees
├── task.toml          # Harbor metadata
├── environment/       # Dockerfile + data/*.log  (agent-visible)
├── solution/          # solve.sh + derive_answer.py + oracle_hints.json  (oracle-only)
└── tests/             # test.sh + test_state.py + expected.json  (verifier-only)

loghub-benchmark/
├── tasks/                       # 180 scored tasks, opaque lh-<hash> ids (60 v1 + 100 v2 + 20 remediation)
├── tracks/tooling/              # 20 unscored log-template-extraction tasks
├── tools/case_builder/          # adapters + exporter + rebuild_tests
├── tools/rubric_check/          # Moonshot rubric grader
├── tests/                       # repo invariants, gameability regressions, snapshots
├── ci_checks/                   # static-check scripts (+ negative fixtures)
├── rubrics/                     # Harbor rubric TOMLs
└── .github/workflows/           # PR validation (static + oracle/nop + rubric)

docs/task-id-map.json maps each opaque id back to its descriptive build slug.

Fuller map: docs/repo-map.md.

See CONTRIBUTING.md. New tasks come from adapter additions or new corpora, not hand-authored single tasks — read docs/dataset-adapters.md for the adapter contract.

This benchmark redistributes small slices (10k–30k lines per task) of log data from the LogPAI Loghub corpus, used here under its license for research and academic work. The Loghub corpus is not BSD/MIT-licensed; it carries a custom research-use license that requires citation and reference to the source repository for any redistribution. See Loghub's LICENSE for the upstream terms.

The benchmark code (adapters, exporter, verifiers, tooling) in this repo is MIT-licensed. The log slices baked into each task's environment/data/ directory are governed by the Loghub license, not MIT.

If you use Loghub-SRE in research or publish results from it, please cite the Loghub corpus paper:

Jieming Zhu, Shilin He, Pinjia He, Jinyang Liu, Michael R. Lyu. Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics. IEEE International Symposium on Software Reliability Engineering (ISSRE), 2023. arXiv:2008.06448

Each dataset has its own original publication that LogPAI asks downstream users to cite. Tasks in this benchmark inherit those obligations:

The full canonical citation text is also reproduced in each task's instruction.md so attribution travels with the data.

• Benchmark code (adapters, exporter, verifiers, tooling, this README): MIT — see LICENSE.
• Log data under tasks/*/environment/data/: governed by the LogPAI Loghub license — free for research and academic work, citation required.
• Per-dataset upstream: each dataset's original publication carries its own license/citation requirements; see the table above.

Commercial uses other than research/evaluation (e.g., redistributing the raw log corpus inside a paid product) require explicit permission from LogPAI and the upstream dataset owners. Open-source benchmark use, internal R&D evaluation, and published comparative results are within scope of the Loghub research-use license.