When Explanations Lie: Stress-Testing Saliency Maps Before Clinical Deployment

A hands-on tutorial for the MICCAI Educational Challenge 2026.

Author: T. Kirscher

This repository contains a Jupyter notebook tutorial on shortcut learning, saliency sanity checks, and trustworthy medical imaging AI. Students train a compact PneumoniaMNIST classifier, implement common saliency methods, and stress-test explanations before considering clinical deployment. The experiment is controlled and pedagogical; it is not a deployable pneumonia detector.

We use simple in-repository PyTorch implementations for educational transparency. For production or large-scale research, users may compare against maintained libraries such as Captum, MONAI, or TorchCAM.

By the end of the tutorial, students will be able to implement saliency methods, explain why plausible heatmaps can mislead, run randomization checks, detect shortcut learning, interpret deletion and stability tests, and connect explanation failures to clinical deployment risks.

Medical image classifiers can appear accurate while relying on scanner marks, borders, site-specific preprocessing, or other non-clinical shortcuts. Saliency maps can help debug such behavior, but they are not proof that a model is safe, fair, or clinically valid.

• notebooks/when_explanations_lie_miccai_2026.ipynb: main tutorial notebook, compatible with Google Colab.
• notebooks/when_explanations_lie_miccai_2026.html: rendered standalone HTML.
• src/: reusable dataset, model, explanation, metric, visualization, and utility code.
• scripts/: validation, notebook execution, and export helpers.
• references/references.bib: BibTeX references used by the tutorial.
• submission/: abstract, AI disclosure, and submission checklist.

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
jupyter lab notebooks/when_explanations_lie_miccai_2026.ipynb

On CPU-only Linux machines, PyPI may download large CUDA-enabled PyTorch wheels. To avoid that, install CPU wheels first:

pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements.txt

Open notebooks/when_explanations_lie_miccai_2026.ipynb and run the cells in order.

python3 scripts/validate_submission.py
python3 scripts/run_notebook.py notebooks/when_explanations_lie_miccai_2026.ipynb

The notebook downloads PneumoniaMNIST automatically through medmnist. A GPU is helpful but not required.

The notebook bootstraps itself in Colab by cloning this repository when it is opened directly from GitHub, then installs any small missing packages.

The rendered tutorial is intended to be served through GitHub Pages at https://kirscher.github.io/when-explanations-lie/. The Pages artifact is built from the committed standalone HTML and adds a small reviewer-facing link bar to the notebook, Colab launch link, repository, and setup instructions.

Runtime depends on hardware, dependency installation, and the first dataset download. The default notebook uses a moderate full-split run; set FAST_DEV_RUN = True for a shorter smoke test.

The tutorial uses PneumoniaMNIST from MedMNIST v2, a lightweight biomedical image classification benchmark. The notebook downloads the dataset automatically through medmnist; no private or restricted data are included in this repository.

• Vanilla gradients
• SmoothGrad
• Integrated Gradients, with a Captum comparison
• Occlusion sensitivity
• Grad-CAM
• Transparent PyTorch implementations of all explanation methods in src/
• Explanation vocabulary: plausibility, faithfulness, localization, and clinical validity
• Model, layer, and label-noise sanity checks
• Shortcut-learning experiment with a synthetic non-clinical marker
• Map correlation, deletion curves, deletion AUC, stability, and shortcut-region attribution
• Learner exercises with expected observations

Attractive heatmaps are not enough for clinical trust. Explanations should be treated as debugging tools and evaluated alongside external validation, clinical metrics, robustness checks, calibration, dataset provenance, and expert review.

If you reuse this tutorial, cite T. Kirscher's repository after public release and cite the papers listed in references/references.bib.

Parts of this tutorial were prepared with assistance from an AI language model for editing, structuring, and clarity. All code, experiments, results, and scientific claims were reviewed and validated by T. Kirscher.

Unless otherwise stated, code is released under the MIT License. Tutorial text and figures are intended for educational reuse with attribution to T. Kirscher.