Reproduce “Friendly Sharpness-Aware Minimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tao Li, Pan Zhou, Zhengbao He, Xinwen Cheng, Xiaolin Huang
Researchers from Shanghai Jiao Tong University and Singapore Management University developed Friendly-SAM (F-SAM), an improved Sharpness-Aware Minimization (SAM) variant that enhances generalization and robustness by discerning and removing detrimental components from SAM's adversarial perturbation. F-SAM achieved up to 0.4% higher accuracy on CIFAR datasets and a 28% gain in robustness to label noise compared to SAM.
Key insights
- The effectiveness of vanilla SAM for improved generalization primarily stems from the batch-specific stochastic gradient noise component within its adversarial perturbation.
- The full gradient component of SAM's adversarial perturbation is detrimental to generalization, with its inclusion often leading to performance degradation.
- Removing the full gradient component and leveraging only the stochastic gradient noise for perturbation improves both generalization and robustness in deep neural networks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.