Reproduce “On the Accuracy of Influence Functions for Measuring Group Effects” on OpenResearch

About this paper

Pang Wei Koh, Kai-Siang Ang, Hubert H. K. Teo, Percy Liang

This research investigates the accuracy of influence functions for quantifying the impact of removing groups of training data, demonstrating that they provide highly correlated rankings of group effects while consistently underestimating their true magnitude across various real-world datasets and group types. The study finds these functions to be practically useful for efficient group-level analysis, despite theoretical limitations for certain evaluation metrics.

Key insights

• Influence functions exhibit surprisingly high Spearman correlation (0.8-1.0) with actual group effects, indicating strong predictive power for ranking group impact despite large group sizes.
• Influence functions consistently underestimate the magnitude of actual effects, often acting as an approximate lower bound, a behavior partly explained by theory for self-loss but not fully for test prediction/loss.
• The one-step Newton approximation is highly accurate, implying that most of the error in influence functions for groups comes from the influence-Newton gap rather than the Newton-actual gap.
• The robust empirical accuracy on real datasets is stronger than current theoretical guarantees, particularly for test prediction/loss, suggesting specific properties of real data enable better performance.

Summary by alphaXiv.