nerbench

A benchmark harness and leaderboard for zero-shot, open-type named-entity recognition.

Benchmark GLiNER-family models and prompted LLMs against many NER datasets under a single canonical schema. Every run is scored with all four SemEval'13 matching schemes (via nervaluate), carries a bootstrap 95% confidence interval and a provenance fingerprint, and rolls up into a mean-rank leaderboard with a self-contained HTML dashboard.

• Backends: gliner: (any GLiNER checkpoint), gliner2: (fastino GLiNER2), llm: (any provider via LiteLLM — Anthropic / OpenAI / Gemini / local Ollama).
• Fair by construction: YAML plans scope each model to the datasets it should run (English-only models aren't averaged over 16 languages).
• Honest about noise: per-cell CIs make sub-noise gaps visible; raw predictions are saved so metrics can be recomputed without re-inference.

uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python -e ".[all]"

Extras: gliner (pulls torch), gliner2, llm (LiteLLM — set the provider's key, e.g. ANTHROPIC_API_KEY / OPENAI_API_KEY), leaderboard (Gradio), dev (pytest), or all. (Not yet on PyPI — install from source.)

# List datasets, suites, and model spec formats
nerbench list

# GLiNER checkpoint on one dataset (quick check with --limit)
nerbench eval --model gliner:urchade/gliner_small-v2.1 --datasets crossner_politics --limit 50

# Full GLiNER-paper OOD benchmark (CrossNER x5 + MIT Movie/Restaurant)
nerbench eval --model gliner:urchade/gliner_multi-v2.1 --suite ood7

# Prompted LLM baseline via LiteLLM (structured outputs; tracks token cost per run)
nerbench eval --model llm:claude-opus-4-8 --datasets crossner_politics --limit 100

# Compare all stored runs (latest per model+dataset, sorted by mean rank)
nerbench compare                    # strict F1
nerbench compare --scheme partial

# Self-contained HTML dashboard (tracks by role/adapter/language, hover for CIs)
nerbench report --open

Instead of running models one at a time, declare a scoped matrix and run it:

nerbench run configs/gliner.yaml --dry-run   # preview the model × dataset matrix
nerbench run configs/gliner.yaml             # execute (writes one JSON per cell)

A plan maps model globs to suites/datasets so specialists and multilingual models only run where it's fair. See configs/ for gliner, gliner2, and llm examples.

Every run writes one JSON to results/<model>/<dataset>/<timestamp>.json with: model metadata, all four schemes (strict, exact, partial, ent_type) each with entity counts and a bootstrap f1_ci, efficiency (wall time, examples/s, LLM token/USD cost), and an env block (harness git SHA + package versions). A companion <timestamp>.pred.jsonl stores the raw gold/predicted spans per example (gitignored by default; --no-predictions to skip).

• nerbench/types.py — canonical schema. Char offsets everywhere (text[start:end] is the entity); every dataset converts once at load time.
• nerbench/predictors/ — the pluggable backend interface: predict(texts, labels) -> list[list[Span]]. Labels are handed over at inference time only — that's the zero-shot contract.
  • gliner.py — any GLiNER-family HF checkpoint (GLiNER, NuNER-zero, ...).
  • gliner2.py — fastino GLiNER2 (model_type=extractor); its own gliner2 package and label-grouped extract_entities API, so a separate backend.
  • llm.py — any provider via LiteLLM (llm:claude-..., llm:openai/..., llm:gemini/..., llm:ollama/...), JSON-schema-constrained output (entity label is an enum of the dataset's label set), surface-string → offset alignment, and token/cost accounting from LiteLLM's model map.
• nerbench/datasets/ — loaders normalize everything to the canonical schema: CrossNER (5 domains), MIT Movie/Restaurant, BC5CDR (any tner/* repo works via the generic loader), and pii_en (ai4privacy pii-masking-300k, streamed, native char offsets). The ood7 suite reproduces the GLiNER-paper OOD benchmark. Terse tags are prettified (politicalparty → political party, BOD → date of birth) so models see natural-language labels.
• nerbench/metrics.py — nervaluate wrapper; stores all four schemes per run (not just one), the raw SemEval entity counts, and a percentile-bootstrap 95% CI on F1 (resampling documents).
• nerbench/evaluate.py — the loop: predict → align → score → one JSON per run, plus the provenance fingerprint and the .pred.jsonl prediction dump.
• nerbench/plan.py — expands a YAML plan (model globs × scoped suites/datasets) into the run matrix; the fairness layer.
• nerbench/report.py — renders results/ into a dependency-free HTML dashboard organized by track (General / Multilingual / Specialists), adapter, and language-family groups; collect() is the leaderboard data contract.
• leaderboard/app.py — Gradio app for an HF Space over the same results/ (WIP).

Aggregation is mean rank (rank per dataset, averaged) so one easy dataset can't dominate the ordering.

• Absolute scores run high vs. published papers (~15–20 F1 pts). We derive gold spans from the same whitespace-joined text fed to the model, so there's no detokenization-mismatch penalty. Relative rankings are trustworthy; absolute cross-paper comparison is not.
• A single cell at --limit 100 is only good to ~±0.05–0.08 F1 (F1 is over entities, not documents). Hover any dashboard score for its CI; the mean-rank ordering across many datasets is far more robust than any one cell.
• Non-whitespace scripts (zh/ja) expose tokenization-frontend fragility that pre-tokenized paper protocols hide.

python -m pytest tests/   # offline: BIO decode, alignment, metrics, report

CI: .github/workflows/tests.yml runs the suite on push; eval.yml is a manually-dispatched eval that commits new result JSONs — the submission path for the leaderboard (no self-reported numbers).

Subclass nerbench.predictors.base.Predictor, implement predict(texts, labels), register it in predictors/registry.py. Backends with API usage implement consume_usage() to feed the cost column.

• Model-vs-model significance (paired bootstrap over shared datasets) — turn the ranking into "A beats B" claims, now that entity counts are stored.
• Per-entity-type breakdown (nervaluate results_by_tag) for error analysis.
• Native-tokenization scoring option to close the absolute-vs-paper gap.
• Nested/overlapping entities (GENIA/ACE) — a class GLiNER can't currently do.
• Per-run wall-clock timeout; full-split reference runs.
• Deploy leaderboard/ as an HF Space reading a git checkout of results/.