Enhancement: calibration diagnostics tailored to discrete estimators (reliability diagram / non-randomized PIT) #23

Context

Likelihood-SBC ranks the true observation's log_prob among synthetic draws' log_probs and tests the ranks for uniformity. For discrete estimators (DiscreteNLE, categorical/count data) the density value is tied across many synthetic draws that equal the true count. PR #7 fixes the immediate correctness problem by switching the rank to a randomized PIT (#(<) + Uniform{0..#(==)}), which is uniform under calibration for discrete predictive distributions (Czado, Gneiting & Held, 2009, Predictive Model Assessment for Count Data, Biometrics). Before that fix, the strict-< rank was systematically left-biased and made calibrated discrete estimators falsely fail SBC.

So rank-SBC is now well-defined for discrete data. This issue is about whether it is the best calibration diagnostic for discrete dimensions, and tracks nicer alternatives. Not a bug — the current behaviour is correct. Post-talk follow-up.

Why revisit for discrete estimators

• The randomized PIT introduces mild RNG dependence (the KS statistic depends on the tie-breaking key).
• Ranking by log_prob value is a highest-density-style transform: valid, but blind to miscalibrations that rearrange mass while preserving the distribution of density values (a power limitation, not a definedness one).
• For discrete/count data there are more direct and interpretable calibration checks.

Options to consider

1. Total Variation on the PMF — already implemented for discrete estimators in PR add discrete nle: count-made, zinb-made #7 (predicted vs. empirical PMF). Cleanest discrete goodness-of-fit; no randomization. Candidate to be the primary discrete calibration signal, with SBC as corroboration.
2. Non-randomized PIT (Czado et al. 2009) — deterministic PIT-reliability curve; removes the RNG dependence of the randomized rank and gives a reliability-diagram view.
3. Reliability diagram / calibration curve — for categorical predictions, binned predicted probability vs. observed frequency. Standard and interpretable for discrete/probabilistic-classification calibration.

Suggested direction

• Keep the randomized-PIT rank fix (correct + standard) but document that discrete estimators lean primarily on TV, and
• add a non-randomized PIT / reliability-diagram check for discrete estimators as an optional, RNG-free calibration view.

References

• Czado, Gneiting & Held (2009), Predictive Model Assessment for Count Data, Biometrics — randomized & non-randomized PIT.
• Talts, Betancourt, Simpson, Vehtari & Gelman (2018), Validating Bayesian Inference Algorithms with Simulation-Based Calibration.

Related: PR #7 (add-discrete-nle) — randomized tie-breaking in setu/validation/_sbc.py.