Model Interpretability with Integrated Gradients

< 목차 >

• Motivation
• Ingegrated Gradients (IG)
• Various Explanation Methods
  • Generic Attention-based Explanation Methods
  • Transformer-based Explanation Methods
  • Gradient-based Attribution Method
• References

Motivation

Ingegrated Gradients (IG)

Integrated Gradients (IG) 란 무엇일까요?

Fig.

Fig.

Fig.

Fig.

Various Explanation Methods

Rethinking Attention-Model Explainability through Faithfulness Violation Test 라는 논문에서는 모델이 왜그런 결과를 냈는지를 해석하는 걸 IG를 포함한 다양한 기존의 방법론들로 비교해봤습니다.

Generic Attention-based Explanation Methods

• Inherent Attention Explanation (RawAtt)
• Attention Gradient (AttGrad)
• Attention InputNorm (AttIN)

Transformer-based Explanation Methods

• Partial Layer-wise Relevance Propagation (PLRP)
• Rollout
• Transformer Attention Attribution (TransAtt)
• Generic Attention Attribution (GenAtt)

Gradient-based Attribution Method

• Input Gradient (InputGrad)
• Integrated Gradients (IG)

References

• Papers
  • Axiomatic Attribution for Deep Networks
  • Rethinking Attention-Model Explainability through Faithfulness Violation Test
  • What Does BERT Look at? An Analysis of BERT’s Attention
• Others (Blogs and Videos)
  • Explaining Machine Learning Models (Talk Slides)
  • Feature Attribution, Stanford CS224U Natural Language Understanding, Spring 2021
  • How to Explain the Prediction of a Machine Learning Model? from lilian weng
  • Why you need to explain machine learning models
• Libraries
  • pytorch/captum
  • Model Interpretability with Captum - Narine Kokhilkyan