Reproduce “Perturbation Sensitivity Analysis to Detect Unintended Model Biases” on OpenResearch

About this paper

Vinodkumar Prabhakaran, Ben Hutchinson, Margaret Mitchell

Researchers from Google Brain introduced Perturbation Sensitivity Analysis (PSA), a general evaluation framework to detect unintended biases in NLP models towards real-world named entities by observing prediction changes when names are perturbed. This method requires no new annotations and demonstrated that both sentiment and toxicity models exhibit notable sensitivity, with up to 40% of classifications changing solely due to name perturbation.

Key insights

• NLP models demonstrate significant and measurable sensitivity to named entity perturbations, indicating that they inadvertently encode social biases related to specific individuals or professions from their training data.
• The degree of model sensitivity is dependent on both the specific names perturbed and the characteristics of the corpus being evaluated, highlighting varying susceptibility to bias across contexts.
• Bias can lead to substantial changes in classification outcomes, even for sentences with moderate or few signals, underscoring the practical implications for fairness in real-world applications.

Summary by alphaXiv.