Reproduce “SMAT: Simple and Efficient Merge-Aware Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yanggan Gu, Yuanyi Wang, Zhen Li, Shuo Cai, Yuhang Liu, Junzhuo Li, Zihao Wang, Hongxia Yang
The authors introduce SMAT, a method that prepares separately fine-tuned experts for later parameter merging. It trains each expert with simulated changes to its update: scaling, masking coordinates, and adding noise as a proxy for other experts’ updates. Across four tested backbones, SMAT raises five-merger average scores over the strongest baselines, with 0.2–1.9% training-time overhead and 1.7–24.3% higher peak GPU memory.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.