Reproduce “The Devil is in the Neurons: Interpreting and Mitigating Social Biases in Pre-trained Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yan Liu, Yu Liu, Xiaokang Chen, Pin-Yu Chen, Daoguang Zan, Min-Yen Kan, Tsung-Yi Ho
Researchers introduced Integrated Gap Gradients (IG2) to pinpoint "Social Bias Neurons" within pre-trained language models, enabling a training-free debiasing method called Bias Neuron Suppression (BNS). BNS achieves a superior balance of fairness and language modeling ability, exemplified by higher Idealized CAT (ICAT) scores, and reveals that current debiasing strategies often relocate bias-contributing neurons to less influential model layers rather than eliminating them.
Key insights
- Integrated Gap Gradients (IG2) effectively pinpoints individual neurons that causally influence social biases, allowing for direct verification through suppression or amplification experiments.
- Traditional debiasing methods like retraining on anti-stereotypical data do not eliminate social bias neurons but rather shift their concentration from deep, impactful layers to shallower, less influential ones.
- Social bias neurons exhibit high consistency for a given unfair judgment across prompt templates but show low overlap across different unfair judgments, indicating precise and targeted bias manifestation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.