Reproduce “What Breaks and What Comes Back: Capability Damage and Recovery in Depth-Pruned Small Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Piyush Dhoka, Vedant Pandhare, Varun Inamdar, Aadarsh Pathre, Pavitha Nooji
The authors test whether pilot-based healing can restore capabilities lost when models are pruned. They remove Transformer blocks and fine-tune the models, finding function-calling and math retention near zero in four families. With a 1M-token final-heal budget, RAH-capped raises OLMo-2-1B math retention from 0.21 to 0.35. At 4.48M tokens, uniform healing has higher point estimates; runtime and inference speed are unmeasured.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.