Reproduce “XAI for Transformers: Better Explanations through Conservative Propagation” on OpenResearch

About this paper

Ameen Ali, Thomas Schnake, Oliver Eberle, Grégoire Montavon, Klaus-Robert Müller, Lior Wolf

Researchers from Tel-Aviv University and Technische Universität Berlin found that gradient-based XAI methods for Transformers suffer from relevance conservation breakdowns in Attention Heads and LayerNorm. They developed modified Layer-wise Relevance Propagation rules, achieving superior explanation quality and accuracy across diverse NLP and graph tasks.

Key insights

• Theoretical analysis demonstrates that Gradient Input (GI) breaks relevance conservation in Attention Heads due to covariance between softmax inputs and input tokens, and in LayerNorm due to the input dependence of variance.
• Explicitly enforcing conservation in these problematic Transformer layers through specialized LRP rules leads to more reliable and faithful explanations that accurately attribute model outputs to input features.
• The proposed approach provides a robust framework for identifying and mitigating biases in complex Transformer models by offering clear, detailed insights into how specific input features influence predictions without relying on unstable counterfactual examples.

Summary by alphaXiv.