IndicPHI-Bench

Evaluation harness and reproduction kit for PII/PHI redaction on Indic clinical text, across 23 Indian languages and 13 scripts.

Companion to the IndicPHI generation pipeline and the indicphi dataset. This repository is the measurement half: it scores any PII detector on the same documents with the same character-level metrics, and it includes the transfer control that turned out to matter more than the benchmark itself.

Fine-tuning a 237M-parameter Indic-pretrained encoder on IndicPHI reaches 0.45% character leakage on held-out synthetic clinical documents, where every off-the-shelf tool we tested leaks 60–68%.

But the same six encoders that tie on that benchmark (0.34–0.64%) span 0.4%–31.8% recall on real human-written Indic text. The benchmark's best model is not the real-text best; a 2.1× larger variant of the best model scores better on the benchmark and transfers 5.7× worse; and past ~10k training documents, more data improves the benchmark and leaves transfer flat.

Do not select a model on the synthetic number alone. That is the finding this harness exists to make measurable — every experiment below is designed so the two numbers are reported side by side.

All numbers from results/final/results.json; every row maps to a command in REPRODUCE.md. Intervals are 95% bootstrap CIs (2,000 resamples).

3,982 documents, 58,988 gold spans, 23 languages.

¹ scored on a 368-document language-stratified sample; every other tool re-scored on those same documents for that comparison.

Regex has near-perfect precision and still leaks two-thirds: it matches the structured identifiers it has patterns for and is blind to every name, place and demographic field. Presidio has the best off-the-shelf leakage and the worst precision — it over-redacts and under-protects. The purpose-built detector's failure is mostly schema: 51% of PII characters here sit in labels its taxonomy has no name for (ABHA ID, BPL ration card, ASHA worker, village/district geography).

Naamapadam, 11 languages × 400 sentences, 2,564 person entities. Identical BIO head, training data and schedule — only the pretrained backbone differs.

All six sit within 0.34–0.64% on the benchmark and span 80× on real text. The benchmark's best model (IndicBERTv2, 0.34%) is not the real-text best. Three controlled pairs isolate why:

• capacity hurts — MuRIL-large vs MuRIL-base: identical corpus, tokenizer and languages, 2.1× parameters. Better on the benchmark (0.39% vs 0.45%), 5.7× worse on real text. The extra capacity memorised the generator.
• paradigm is irrelevant to transfer — GLiNER vs mDeBERTa-BIO share a backbone and differ only in tagging approach: 6× apart in-distribution, 4.4% vs 4.4% on real text.
• pretraining decides it — Indic (31.8%, 21.4%) > generic multilingual (18.5%, 4.4%)

  non-Indic (0.4%), reproduced across two independent Indic corpora.

Off-the-shelf systems on the same real text: regex 0.0%, GLiNER zero-shot 1.1% [0.7, 1.6], LFM2.5-PII 4.0% [3.2, 4.7], Presidio 7.4% [6.4, 8.5] — every one under 8%.

The final 2.25× of data halved in-distribution error and moved real-text recall by −0.7 points. At 5,000 documents the benchmark already reads as nearly solved while the model finds under 10% of real names.

Training the same MuRIL-base with the same head on real news data instead:

The task is learnable — the architecture reaches 93.4% given real in-domain data, with less of it. What this does not separate is synthetic-vs-real from clinical-vs-news; closing that needs real annotated clinical text (see GOLD_SET_SPEC.md).

• Seed variance exceeds evaluation variance. Retraining at three seeds: MuRIL-base 31.7% ± 2.9, IndicBERTv2 22.5% ± 5.3 — the ranges overlap. We make no claim about ordering within the Indic-pretrained tier; the tier gaps (10–30 points) are what survive.
• The LLM arm is gpt-5-nano only, sampled at its default temperature (it rejects any other), so it is the one non-deterministic system here. Nothing here speaks to frontier models.
• Two scripts — Ol Chiki (Santali) and Meetei Mayek (Manipuri) — are absent from every encoder we could find, and are the weakest languages for every system tested.

Span-level F1 answers "was this entity found with the right label and boundaries?". Redaction asks something different: of all sensitive characters, how many are still readable after the tool ran? A name with one uncovered character is still a leak; a name found but mislabelled is not.

Character-level scoring is what makes cross-tool comparison fair: regex, Presidio, GLiNER and each encoder tokenize differently, so tokens are not a common currency but characters are. Confidence intervals are 2,000-resample bootstraps.

scripts/
  predictors.py          uniform interface — any tool returning character spans
  evaluate.py            core metrics
  analyze.py             full comparison + bootstrap CIs + per-language tables
  eval_naamapadam.py     real-text transfer test (the control)
  eval_gold.py           score on a real annotated gold set (schema below)
  transfer_stats.py      transfer CIs + benchmark/transfer rank correlation
  train_encoder.py       BIO-head fine-tuning, any HF encoder backbone
  train_gliner.py        GLiNER (prompted-span) fine-tuning
  train_ceiling.py       trained ceiling: same architecture on real data
  figures.py             publication figures
  error_analysis.py      failure modes grouped by what they cost
*.sh                     one runner per experiment (see REPRODUCE.md)
results/final/           frozen results every published number derives from
figures/                 generated figures (PDF + PNG)
data/gold/fixture.jsonl  12-document fixture in the gold-set schema

Needs Python ≥3.10 with torch (CUDA), transformers, gliner, presidio-analyzer. run.sh pins one interpreter for everything — set PY if yours differs.

PY=/path/to/python ./run.sh scripts/analyze.py --limit 200 --tools regex,presidio

Adding a tool is one class with a __call__(text) -> [{"start","end","label"}]; see predictors.py. Nothing else needs to change.

REPRODUCE.md maps every table and figure to the exact command, and documents seven failure modes that cost real time — including a trainer that silently skips OOM batches while printing a loss, and a loader that returns a randomly-initialised model without raising.

Not included here: trained checkpoints (~30 GB) and the source corpus. Both are regenerable from REPRODUCE.md, or available from the dataset and model repos.

Every in-distribution number comes from one synthetic generator, and the only real annotated Indic text available to us is news, not clinical — so the transfer result confounds synthetic-vs-real with clinical-vs-news. GOLD_SET_SPEC.md specifies exactly what would close it: 200–500 real annotated clinical documents, enough for tool comparison, not training. eval_gold.py already accepts that format and validates it (offsets, labels, overlaps) before scoring:

./run.sh scripts/eval_gold.py your_gold.jsonl --validate-only

If you have access to real Indic clinical text and can annotate even a few hundred documents, that single contribution would resolve the central limitation of this work.

MIT. The corpus, pipeline and models are released separately under their own terms; note that checkpoints derived from LFM2.5 inherit the LFM1.0 license, not MIT.

Paper in preparation; citation to follow.