Reproduce “The Surprising Effectiveness of Deleting Weights in LLM Post-Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jack Lu, Zhenbang Yang, Mike Lasby, Tejas Pote, Yani Ioannou, Mengye Ren
Deletion Tuning learns masks that zero selected gate-projection weights while keeping the pretrained model frozen. It uses trainable logits initialized from Wanda scores, rather than learning replacement values. On Qwen3-30B-A3B, it reaches 58.54% versus full fine-tuning’s 58.43% on the SimpleRL-Zoo mean, with a 29.6-MB adapter versus 116,471-MB. Retention evidence is limited to GRPO runs on three Llama sizes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.