Reproduce “Evaluating Saliency Methods for Neural Language Models” on OpenResearch

About this paper

Shuoyang Ding, Philipp Koehn

Researchers from Johns Hopkins University developed a quantitative benchmark to evaluate saliency methods for neural language models, assessing interpretation quality via plausibility to human intuition and faithfulness to model behavior. Their findings indicate that increasing plausibility does not assure faithfulness, and interpretation reliability is highly dependent on model architecture and configuration.

Key insights

• SmoothGrad and Integrated Gradients generally offer more plausible interpretations than Vanilla Gradient, but this does not consistently translate to higher faithfulness.
• The quality of saliency interpretations, particularly faithfulness, is highly sensitive to model architecture, with Transformers often exhibiting better plausibility.
• A significant finding is that higher plausibility does not guarantee higher faithfulness, indicating that interpretations can align with human intuition without accurately reflecting the model's internal mechanism.
• Interpretation quality is highly sensitive to even minor model configuration changes and can paradoxically improve with smaller, distilled student models.

Summary by alphaXiv.