Reproduce “Explaining f-Divergence-Based Regularization via Local Curvature and Sharpness-Aware Minimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nour Jamoussi, Marios Kountouris
A local second-order analysis shows that smooth f-divergence regularizers penalize changes in predictive distributions according to Fisher geometry in parameter space and a related pullback geometry in input space. Experiments with asymmetric Jensen–Shannon divergence on four image-classification datasets associate the strongest local coefficient, at α=0.5, with generally better accuracy, lower negative log-likelihood, and flatter measured loss landscapes, while clarifying that the connection to SAM is conditional rather than an equivalence.
Key insights
- Near identical distributions, all smooth f-divergences share the same quadratic structure up to the scalar factor φ''(1); distinctions between divergences therefore emerge mainly through higher-order behavior and global properties.
- Divergence regularization and SAM are curvature-sensitive in different ways: the former uses Fisher-weighted predictive sensitivity, whereas SAM targets the largest Hessian curvature within a parameter neighborhood.
- For α-skew JSD, α=0.5 maximizes the local curvature coefficient, providing a controlled setting for testing whether stronger distributional sensitivity penalties correspond to improved performance and flatter solutions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.