Faithful and Efficient Circuit Discovery via Relevance Patching

Activation patching is a standard method in mechanistic interpretability for localizing the components of a model responsible for specific behaviors, but it is computationally expensive to apply at scale. Attribution patching offers a faster, gradient-based approximation, yet suffers from noise and reduced reliability in deep, highly non-linear networks. In this work, we introduce Relevance Patching (RelP), which replaces the local gradients in attribution patching with propagation coefficients derived from Layer-wise Relevance Propagation (LRP). LRP propagates the network's output backward through the layers, redistributing relevance to lower-level components according to local propagation rules that ensure properties such as relevance conservation or improved signal-to-noise ratio. Like attribution patching, RelP requires only two forward passes and one backward pass, maintaining computational efficiency while improving faithfulness. We validate RelP across a range of models and tasks, showing that it more accurately approximates activation patching than standard attribution patching, particularly when analyzing residual stream and MLP outputs in the Indirect Object Identification (IOI) task. For instance, for MLP outputs in GPT-2 Large, attribution patching achieves a Pearson correlation of 0.006, whereas RelP reaches 0.956, highlighting the improvement offered by RelP. Additionally, we compare the faithfulness of sparse feature circuits identified by RelP and Integrated Gradients (IG), showing that RelP achieves comparable faithfulness without the extra computational cost associated with IG.

git clone https://github.com/FarnoushRJ/RelP.git
cd RelP/TransformerLens
pip install -e .

import transformer_lens

# Load a model (eg GPT-2 Small)
model = transformer_lens.HookedTransformer.from_pretrained("gpt2-small")
model.cfg.use_lrp = True  # Use LRP
model.cfg.LRP_rules = ['LN-rule', 'Identity-rule', 'Half-rule']  # LRP rules to be used

# Run the model and get logits and activations
logits, activations = model.run_with_cache("Hello World")

The following rules are currently implemented and supported.

We build on the TransformerLens library, which provides the foundation for much of this code.

This project is distributed under the Apache 2.0 License. It incorporates code and models from third-party sources, which are provided under their own licenses. Please review those licenses before use.

@misc{jafari2025relp,
      title={RelP: Faithful and Efficient Circuit Discovery via Relevance Patching}, 
      author={Farnoush Rezaei Jafari and Oliver Eberle and Ashkan Khakzar and Neel Nanda},
      year={2025},
      eprint={2508.21258},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2508.21258}, 
}