Reproduce “SeWA: Selective Weight Average via Probabilistic Masking” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Peng Wang, Shengchao Hu, Zerui Tao, Guoxia Wang, Dianhai Yu, Li Shen, Quan Zheng, Dacheng Tao

Selective Weight Averaging (SeWA) automates the process of combining model checkpoints by learning which ones to include, transforming a discrete selection into a continuous optimization problem. This method achieves superior performance and computational efficiency across tasks like behavior cloning, image classification, and text classification, for instance, yielding an average cumulative reward of 3334.907 with K=10 checkpoints in behavior cloning, exceeding LAWA's 3288.249.

Key insights

  • The discrete problem of selecting which model checkpoints to average can be reformulated as a continuous optimization problem using probabilistic masking with Gumbel-Softmax reparameterization.
  • Adaptively learning to select relevant checkpoints leads to improved generalization performance and better computational efficiency compared to fixed averaging strategies.
  • Theoretical analysis demonstrates that SeWA achieves sharper generalization bounds than standard SGD and other averaging methods in both convex and non-convex settings.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.