skip to main content
research-article

Advancing Attribution-Based Neural Network Explainability through Relative Absolute Magnitude Layer-Wise Relevance Propagation and Multi-Component Evaluation

Published: 15 April 2024 Publication History

Abstract

Recent advancement in deep-neural network performance led to the development of new state-of-the-art approaches in numerous areas. However, the black-box nature of neural networks often prohibits their use in areas where model explainability and model transparency are crucial. Over the years, researchers proposed many algorithms to aid neural network understanding and provide additional information to the human expert. One of the most popular methods being Layer-Wise Relevance Propagation (LRP). This method assigns local relevance based on the pixel-wise decomposition of nonlinear classifiers. With the rise of attribution method research, there has emerged a pressing need to assess and evaluate their performance. Numerous metrics have been proposed, each assessing an individual property of attribution methods such as faithfulness, robustness, or localization. Unfortunately, no single metric is deemed optimal for every case, and researchers often use several metrics to test the quality of the attribution maps. In this work, we address the shortcomings of the current LRP formulations and introduce a novel method for determining the relevance of input neurons through layer-wise relevance propagation. Furthermore, we apply this approach to the recently developed Vision Transformer architecture and evaluate its performance against existing methods on two image classification datasets, namely ImageNet and PascalVOC. Our results clearly demonstrate the advantage of our proposed method. Furthermore, we discuss the insufficiencies of current evaluation metrics for attribution-based explainability and propose a new evaluation metric that combines the notions of faithfulness, robustness, and contrastiveness. We utilize this new metric to evaluate the performance of various attribution-based methods. Our code is available at: https://github.com/davor10105/relative-absolute-magnitude-propagation

References

[1]
Samira Abnar and Willem Zuidema. 2020. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 4190–4197. DOI:
[2]
Ameen Ali, Thomas Schnake, Oliver Eberle, Grégoire Montavon, Klaus-Robert Müller, and Lior Wolf. 2022. XAI for transformers: Better explanations through conservative propagation. In Proceedings of the International Conference on Machine Learning, ICML 2022 (Proceedings of Machine Learning Research, Vol. 162). PMLR, Baltimore, Maryland, USA, 435–451. Retrieved from https://proceedings.mlr.press/v162/ali22a.html
[3]
David Alvarez Melis and Tommi Jaakkola. 2018. Towards robust interpretability with self-explaining neural networks. In Advances in Neural Information Processing Systems. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc., Montréal, Canada. Retrieved from https://proceedings.neurips.cc/paper/2018/file/3e9f0fc9b2f89e043bc6233994dfcf76-Paper.pdf
[4]
David Alvarez-Melis and Tommi S. Jaakkola. 2018. On the robustness of interpretability methods. arXiv:1806.08049. Retrieved from http://arxiv.org/abs/1806.08049
[5]
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. 2018. Towards Better Understanding of Gradient-Based Attribution Methods for Deep Neural Networks. In Proceedings of the International Conference on Learning Representations . Retrieved from https://openreview.net/forum?id=Sy21R9JAW
[6]
Anna Arias-Duart, Ferran Parés, Dario Garcia-Gasulla, and Victor Giménez-Ábalos. 2022. Focus! rating XAI methods and finding biases. In Proceedings of the 2022 IEEE International Conference on Fuzzy Systems. IEEE, FUZZ-IEEE, Padua, Italy, 1–8.
[7]
Leila Arras, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. 2017. Explaining recurrent neural network predictions in sentiment analysis. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. Association for Computational Linguistics, Copenhagen, Denmark, 159–168. DOI:
[8]
Leila Arras, Ahmed Osman, and Wojciech Samek. 2020. Ground truth evaluation of neural network explanations with clevr-XAI. Information Fusion 81, 3 (2020), 14–40. DOI:
[9]
Vijay Arya, Rachel K. E. Bellamy, Pin-Yu Chen, Amit Dhurandhar, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Q. Vera Liao, Ronny Luss, Aleksandra Mojsilovic, Sami Mourad, Pablo Pedemonte, Ramya Raghavendra, John T. Richards, Prasanna Sattigeri, Karthikeyan Shanmugam, Moninder Singh, Kush R. Varshney, Dennis Wei, and Yunfeng Zhang. 2019. One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques. arXiv:1909.03012. Retrieved from http://arxiv.org/abs/1909.03012
[10]
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. 2015. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE 10, 7 (July 2015), e0130140. DOI:
[11]
Umang Bhatt, Adrian Weller, and José M. F. Moura. 2020. Evaluating and aggregating feature-based model explanations. In Proceedings of the 29th International Joint Conference on Artificial Intelligence. IJCAI, Yokohama, Japan, 3016–3022. DOI:
[12]
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In Proceedings of the Computer Vision - ECCV 2020-16th European Conference. Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.), Lecture Notes in Computer Science, Vol. 12346 Springer, Glasgow, UK, 213–229. DOI:
[13]
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N. Balasubramanian. 2018. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision. IEEE, IEEE Workshop on Applications of Computer Vision, Lake Tahoe, NV, USA, 839–847.
[14]
Hila Chefer, Shir Gur, and Lior Wolf. 2021. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021. Computer Vision Foundation / IEEE, Virtual Event, 782–791. DOI:
[15]
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. 2021. Decision transformer: Reinforcement learning via sequence modeling. In Proceedings of the Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021. NeurIPS, Virtual Event, 15084–15097. Retrieved from https://openreview.net/pdf?id=a7APmM4B9d
[16]
Sanjoy Dasgupta, Nave Frost, and Michal Moshkovitz. 2022. Framework for evaluating faithfulness of local explanations. In Proceedings of the Machine Learning Research. Baltimore, Maryland, USA, 4794–4815. DOI:
[17]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. Ieee, Conference on Computer Vision and Pattern Recognition, Miami, Florida, USA, 248–255.
[18]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Volume 1. Association for Computational Linguistics, Minneapolis, MN, USA, 4171–4186. DOI:
[19]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, Virtual Event, Austria. Retrieved from https://openreview.net/forum?id=YicbFdNTTy
[20]
Rachel Lea Draelos and Lawrence Carin. 2021. Use HiResCAM instead of Grad-CAM for faithful explanations of convolutional neural networks. arxiv:2011.08891. Retrieved from https://arxiv.org/abs/2011.08891
[21]
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. 2010. The pascal visual object classes challenge. International Journal of Computer Vision 88, 2 (June 2010), 303–338.
[22]
Jindong Gu, Yinchong Yang, and Volker Tresp. 2019. Understanding individual decisions of CNNs via contrastive backpropagation. In Proceedings of the Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision. Springer, ACCV, Perth, Australia, 119–134.
[23]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. CVPR, Las Vegas, Nevada, 770–778.
[24]
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In Proceedings of the 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3–7, 2021. OpenReview.net, Virtual Event, Austria. Retrieved from https://openreview.net/forum?id=XPZIaotutsD
[25]
Peng-Tao Jiang, Chang-Bin Zhang, Qibin Hou, Ming-Ming Cheng, and Yunchao Wei. 2021. Layercam: Exploring hierarchical class activation maps for localization. IEEE Transactions on Image Processing 30, 1 (2021), 5875–5888.
[26]
Yiming Ju, Yuanzhe Zhang, Zhao Yang, Zhongtao Jiang, Kang Liu, and Jun Zhao. 2022. Logic traps in evaluating attribution scores. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Dublin, Ireland, 5911–5922.
[27]
Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017 (Proceedings of Machine Learning Research, Vol. 70). PMLR, Sydney, NSW, Australia, 1885–1894. Retrieved from http://proceedings.mlr.press/v70/koh17a.html
[28]
Maximilian Kohlbrenner, Alexander Bauer, Shinichi Nakajima, Alexander Binder, Wojciech Samek, and Sebastian Lapuschkin. 2020. Towards best practice in explaining neural network decisions with LRP. In Proceedings of the 2020 International Joint Conference on Neural Networks. IEEE, IJCNN, Glasgow, United Kingdom, 1–7.
[29]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of the Advances in Neural Information Processing Systems, F. Pereira, C. J. Burges, L. Bottou, and K. Q. Weinberger (Eds.), Vol. 25. Curran Associates, Inc., Lake Tahoe, Nevada, USA. DOI:
[30]
Junhee Lee, Hyeonseong Cho, Yun Jang Pyun, Suk Ju Kang, and Hyoungsik Nam. 2022. Heatmap assisted accuracy score evaluation method for machine-centric explainable deep neural networks. IEEE Access 10, 1 (2022), 64832–64849. DOI:
[31]
Yibing Liu, Haoliang Li, Yangyang Guo, Chenqi Kong, Jing Li, and Shiqi Wang. 2022. Rethinking attention-model explainability through faithfulness violation test. In Proceedings of the International Conference on Machine Learning. PMLR, ICML, 1 W Pratt St, Baltimore, MD 21201, USA, 13807–13824.
[32]
Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. 2018. Methods for interpreting and understanding deep neural networks. Digital Signal Processing 73, 2 (february 2018), 1–15. DOI:
[33]
Woo-Jeoung Nam, Shir Gur, Jaesik Choi, Lior Wolf, and Seong-Whan Lee. 2020. Relative attributing propagation: Interpreting the comparative contributions of individual units in deep neural networks. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, AAAI 2020, The 32nd Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The 10th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7–12, 2020. AAAI Press, New York, NY, USA, 2501–2508. http://arxiv.org/abs/1904.00605
[34]
An-phi Nguyen and María Rodríguez Martínez. 2020. On quantitative aspects of model interpretability. arXiv:2007.07584. Retrieved from https://arxiv.org/abs/2007.07584
[35]
Vitali Petsiuk, Abir Das, and Kate Saenko. 2018. RISE: Randomized input sampling for explanation of black-box models. In Proceedings of the British Machine Vision Conference 2018, BMVC 2018. BMVA Press, Newcastle, UK, 151. Retrieved from http://bmvc2018.org/contents/papers/1064.pdf
[36]
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image generation. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021 (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, Virtual Event, 8821–8831. Retrieved from http://proceedings.mlr.press/v139/ramesh21a.html
[37]
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. “Why Should I Trust You?” Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, San Francisco, CA, USA, 1135–1144. DOI:
[38]
Laura Rieger and Lars Kai Hansen. 2020. IROF: A low resource evaluation metric for explanation methods. arXiv:2003.08747. Retrieved from https://arxiv.org/abs/2003.08747
[39]
Yao Rong, Tobias Leemann, Vadim Borisov, Gjergji Kasneci, and Enkelejda Kasneci. 2022. A consistent and efficient evaluation strategy for attribution methods. In Proceedings of the Machine Learning Research. PMLR, Baltimore, Maryland, USA, 18770–18795. arXiv:arXiv:2202.00449
[40]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet large scale visual recognition challenge. International Journal of Computer Vision 115, 3 (April 2015), 211–252. DOI:
[41]
Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. 2016. Evaluating the visualization of what a deep neural network has learned. IEEE Transactions on Neural Networks and Learning Systems 28, 11 (2016), 2660–2673.
[42]
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2019. Grad-CAM: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision 128, 2 (Octiober 2019), 336–359. DOI:
[43]
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. 2016. Not just a black box: Learning important features through propagating activation differences. arXiv:1605.01713. Retrieved from http://arxiv.org/abs/1605.01713
[44]
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. 2016. Mastering the game of Go with deep neural networks and tree search. Nature 529, 7587 (January 2016), 484–489. DOI:
[45]
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. In 2nd International Conference on Learning Representations, ICLR 2014, Yoshua Bengio and Yann LeCun (Eds.). ICLR, Banff, AB, Canada. http://arxiv.org/abs/1312.6034
[46]
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. 2017. SmoothGrad: Removing noise by adding noise. arXiv:1706.03825. Retrieved from http://arxiv.org/abs/1706.03825
[47]
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017 (Proceedings of Machine Learning Research, Vol. 70). PMLR, Sydney, NSW, Australia, 3319–3328. Retrieved from http://proceedings.mlr.press/v70/sundararajan17a.html
[48]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017. NeurIPS, Long Beach, CA, USA, 5998–6008. Retrieved from https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
[49]
Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harvard Journal of Law and Technology 31, 2 (2017), 841.
[50]
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015 (JMLR Workshop and Conference Proceedings, Vol. 37). JMLR.org, Lille, France, 2048–2057. Retrieved from http://proceedings.mlr.press/v37/xuc15.html
[51]
Chih-Kuan Yeh, Cheng-Yu Hsieh, Arun Suggala, David I. Inouye, and Pradeep K. Ravikumar. 2019. On the (in) fidelity and sensitivity of explanations. Advances in Neural Information Processing Systems 32, 1 (2019), 10967–10978.
[52]
Matthew D. Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In Proceedings of the Computer Vision - ECCV 2014-13th European Conference, Lecture Notes in Computer Science, Vol. 8689. Springer, Zurich, Switzerland, 818–833. DOI:
[53]
Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. 2018. Top-down neural attention by excitation backprop. International Journal of Computer Vision 126, 10 (2018), 1084–1102.

Cited By

View all
  • (2025)GradToken: Decoupling tokens with class-aware gradient for visual explanation of Transformer networkNeural Networks10.1016/j.neunet.2024.106837181(106837)Online publication date: Jan-2025

Index Terms

  1. Advancing Attribution-Based Neural Network Explainability through Relative Absolute Magnitude Layer-Wise Relevance Propagation and Multi-Component Evaluation

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Transactions on Intelligent Systems and Technology
      ACM Transactions on Intelligent Systems and Technology  Volume 15, Issue 3
      June 2024
      646 pages
      EISSN:2157-6912
      DOI:10.1145/3613609
      • Editor:
      • Huan Liu
      Issue’s Table of Contents

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 15 April 2024
      Online AM: 26 February 2024
      Accepted: 03 February 2024
      Revised: 18 January 2024
      Received: 19 June 2023
      Published in TIST Volume 15, Issue 3

      Check for updates

      Author Tags

      1. Explainable artificial intelligence
      2. Vision Transformer
      3. layer-wise relevance propagation
      4. attribution-based evaluation

      Qualifiers

      • Research-article

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)241
      • Downloads (Last 6 weeks)17
      Reflects downloads up to 12 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2025)GradToken: Decoupling tokens with class-aware gradient for visual explanation of Transformer networkNeural Networks10.1016/j.neunet.2024.106837181(106837)Online publication date: Jan-2025

      View Options

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      Full Text

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media