Reproduce “EMASAM: a Computationally Efficient Sharpness-Aware Minimization via EMA-Guided Perturbations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tanapat Ratchatorn, Masayuki Tanaka
EMASAM, an efficient variant of Sharpness-Aware Minimization (SAM), uses Exponential Moving Average (EMA)-guided perturbations to reduce computational cost. The method matches or exceeds the generalization performance of SAM across various architectures and datasets, while requiring only a single backward pass per training iteration, effectively halving the computational overhead.
Key insights
- An EMA of model parameters can serve as a stable and computationally efficient reference point for generating sharpness-aware perturbations.
- Perturbing the model based on its current state relative to a smoothed EMA state effectively proxies a 'worst-case' scenario, guiding the model to flatter minima without explicit, costly gradient calculations.
- Decoupling the perturbation mechanism from instantaneous mini-batch gradients improves the stability and consistency of the perturbation direction during training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.