Reference numbers for bayes-hdc. The calibration / selective / OOD numbers come from make bench; the canonical-dataset and paper tables come from make bench-canonical and make bench-paper (separate targets because they need the optional torch / torch-hd / gdown extras and a network connection on first run). Figures are checked into benchmarks/figures/.

Reproduce:

make install-all
make bench                # calibration/selective/OOD (~2 minutes, laptop CPU)
make bench-canonical      # ISOLET/UCI-HAR/EMG vs TorchHD (downloads data on first run)
make bench-paper          # classification calibration + multi-seed anomaly
make figures              # regenerate the figures under benchmarks/figures/
# or, containerised:
make docker-bench         # writes to benchmarks/results/

Each results JSON under benchmarks/ embeds its own provenance (commit, Python / JAX / TorchHD versions, platform). Canonical and paper tables last refreshed June 10, 2026 on Python 3.13.5, JAX 0.10.0, torchhd 5.8.4, macOS arm64; the committed benchmarks/canonical_results.json and benchmarks/paper_results.json are the exact artifacts behind the tables below and in the paper.

The numbers below are not a like-for-like single-classifier comparison: the Bayes-HDC column reports the best of an ensemble of four classifiers (ridge regression on hypervectors, logistic regression on hypervectors, centroid-LVQ matching TorchHD's default, and an off-the-shelf gradient- boosted baseline on raw features), with the per-task winner chosen by held-out calibration-set accuracy. The TorchHD column reports its default centroid classifier on the same encoder. The setup is closer to "how well does the ensemble + cal-set selection do" than to "is bind/bundle in JAX faster than in PyTorch"; read the deltas accordingly.

Numbers are single-seed; the JSON dump in benchmarks/benchmark_calibration_results.json records exact configurations. A multi-seed sweep is a planned addition.

Both libraries use the same TemperatureCalibrator (L-BFGS in log-space), isolating the effect of the underlying classifier's logit distribution.

Split-conformal APS (Romano et al. 2020) at α = 0.1 target (≥ 0.90 coverage):

All datasets clear the finite-sample coverage guarantee. Set size scales with task difficulty — binary classification collapses sets to near-1, 10-class problems admit 2–3 classes.

Reproduced from benchmarks/benchmark_compare.py on the same machine, identical dimensions (d = 10 000), 20 warmup iterations, 200 timed trials. Both libraries run on CPU. JAX side compiles via jit once during warmup; PyTorch side uses TorchHD's default eager kernels under torch.no_grad. Both timers wait for the result tensor on-device — no asymmetric host sync via .item() — so the numbers measure "compute-and-block" on each backend.

Pointwise operations (bind, bundle, cosine) are 1.4×–3.5× faster under JAX-jit than TorchHD's eager kernels. The encoder benchmark includes 200 random-sample lookups in addition to the bind+bundle work, and TorchHD's embeddings.Random indexing path stays slightly faster on this single-CPU configuration; the gap typically closes on GPU and reverses with batching.

These numbers compare eager-mode TorchHD against jit-compiled bayes-hdc. A torch.compile baseline would partially close the pointwise gap and is deferred to a future suite. Reproduce locally with python benchmarks/benchmark_compare.py; the script writes benchmarks/benchmark_results.json (gitignored — local hardware varies).

Per-position retrieval accuracy at fixed d = 4 096, codebook size 256, chunk size 16, averaged over 3 seeds. Each item is drawn i.i.d. from the codebook; retrieval applies get(i) then cleanup against the codebook (argmax cosine similarity).

Flat permute-bundle saturates around T ≈ 200 and degrades monotonically as T grows: by T = 800 the flat representation retrieves only 31 % of positions correctly. The hierarchical variant stays at perfect retrieval throughout the swept range, because the chunk-level cleanup (an argmax against the cached chunk codebook) prunes the cross-chunk noise before the inner un-permute. At chunk_size = 16, both layers carry only O(√T) items, and the per-layer SNR is dominated by 1/√(chunk_size) and 1/√(n_chunks) rather than 1/√T. Reproduce with python benchmarks/benchmark_sequence_capacity.py. References: Plate (2003) §6.2 on flat-bundle capacity; Frady, Kleyko & Sommer (2018) on hierarchical recurrent-network indexing.

The accuracy table above uses sklearn datasets (iris / wine / breast-cancer / digits / MNIST) — useful as smoke checks, not as HDC-canonical anchors. The table below anchors on the datasets the HDC literature actually benchmarks on, head-to-head against a TorchHD centroid over its Sinusoid embedding (the matching random-Fourier encoder family) on identical splits.

Protocol: the training pool is split 70/15/15 into fit / model-selection / conformal-calibration slices. Both encoders get the same RBF-bandwidth grid search on the model-selection slice (the grid includes TorchHD's untuned default), and the conformal + temperature calibration uses only the disjoint calibration slice, so the finite-sample coverage guarantee is intact. Test data is never touched before final evaluation. Mean ± std over 5 seeds, d = 10 000, α = 0.1.

Under a matched tuning budget bayes-hdc is ahead on ISOLET and EMG and behind on UCI-HAR — competitive with the deterministic reference, not dominant. The value-add is the probabilistic layer: calibrated probabilities (order-of-magnitude ECE reduction) and conformal coverage at the target, guarantees the deterministic baseline does not provide. Note the EMG split is window-level (subjects shared across train/test, no canonical split ships with the dataset); UCI-HAR uses the official subject-disjoint partition, so its accuracies are comparable to the literature.

Reproduce with:

uv run --with scikit-learn --with torch --with torch-hd --with gdown \
    python benchmarks/benchmark_canonical.py

ISOLET is fetched via TorchHD's dataset loader (canonical isolet1-4/isolet5 split); UCI-HAR via bayes_hdc.datasets.load_ucihar, which downloads the official UCI archive and keeps the subject-disjoint 7352/2947 partition; EMG via bayes_hdc.datasets.load_emg, which downloads the original authors' dataset.mat from abbas-rahimi/HDC-EMG and cuts label-pure 256-sample windows. European Languages (Joshi, Halseth, Kanerva 2016) remains future work: the corpus has no stable OpenML/UCI mirror and needs the Wortschatz / Europarl preprocessing pipeline.

These are head-to-head comparisons the library will gain once the matching datasets / configurations are available on the dev machine. Documented here so they are not silently absent from the benchmarks story.

• ConformalHDC (Liang, Poursiami, Yang, Cooper, Jaiswal, Parsa, Fortin & Shahbaba 2026, arXiv:2602.21446). Concurrent algorithmic work on conformal prediction for HDC; cited in the README, paper, and bibliography. A like-for-like comparison on their reported hippocampal-neural-decoding dataset would cash out the "concurrent algorithmic priority; library-first implementation" framing into hard numbers (e.g. matching their adaptive-score coverage within 1-2 pp at comparable set sizes). Deferred to the next benchmark cycle.
• torch.compile head-to-head with TorchHD. The Wall-clock table above compares eager-mode TorchHD against jit-compiled bayes-hdc; a torch.compile baseline would partially close the pointwise gap and isolate the JAX-XLA-vs-PyTorch-compile contribution from the JIT-vs-eager contribution.
• GPU + TPU numbers. All wall-clock measurements above are single-CPU. Multi-device numbers via pmap_* and shard_map_* wrappers are runnable but not yet reported on a TPU pod with multiple chips.

The commit includes paper-ready figures under benchmarks/figures/ — 10 PNGs at 150 DPI (PDF versions regenerable via make figures).

Dataset	Reliability
iris
wine
breast-cancer
digits

Dataset	Coverage
iris
wine
breast-cancer
digits