Reproduce “Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization” on OpenResearch

About this paper

Simone Bombari, Marco Mondelli

This research statistically characterizes spurious correlations in high-dimensional regression, demonstrating how explicit regularization and implicit 'effective regularization' from over-parameterization, alongside data structure, drive models to learn non-causal patterns. It mathematically formalizes a trade-off where minimizing in-distribution test loss can increase spurious correlation learning.

Key insights

• Unregularized linear models (when sample size sufficiently exceeds dimension) do not inherently learn spurious correlations, unlike over-parameterized models.
• The 'simplicity' of spurious features (via `λ_max(Σ_yy)`) and their statistical correlation with core features (`λ_min(S_Σ_x)`) are critical determinants of the extent of learned spurious correlations.
• Over-parameterized models, such as Random Features models, exhibit an intrinsic 'effective regularization' that is strictly positive even without explicit penalties, explaining why they tend to learn spurious features.
• A fundamental trade-off exists where minimizing in-distribution test loss can lead to increased spurious correlation learning, despite its negative impact on out-of-distribution performance.

Summary by alphaXiv.