Reproduce “Neuron Shapley: Discovering the Responsible Neurons” on OpenResearch

About this paper

Amirata Ghorbani, James Zou

Neuron Shapley introduces a theoretically principled framework using Shapley values to quantify the contribution of individual neurons in deep neural networks, making it computationally feasible for large models via the Truncated Multi-Armed Bandit Shapley (TMAB-Shapley) algorithm. This method precisely identifies neurons responsible for overall performance, class-specific predictions, unfair biases, and adversarial vulnerabilities, enabling rapid, post-training model repair to enhance fairness and robustness.

Key insights

• TMAB-Shapley achieves orders of magnitude speed-up over standard Monte-Carlo Shapley estimation while maintaining high accuracy in identifying top-k important neurons.
• A remarkably sparse set of neurons are disproportionately critical for specific network behaviors, including overall accuracy, class-specific predictions, fairness, and robustness, suggesting a 'lottery ticket' phenomenon at the neuron level.
• Neuron Shapley can function as a 'diagnostic tool' and 'scalpel' for targeted model repair, allowing post-training modifications (zeroing out specific neurons) to mitigate bias or adversarial vulnerabilities without requiring extensive retraining.

Summary by alphaXiv.