Reproduce “On the Limits of Layer Pruning for Generative Reasoning in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Safal Shrestha, Anubhav Shrestha, Aadim Nepal, Minwu Kim, Keith Ross
Researchers at New York University Abu Dhabi characterized the limits of layer pruning for generative reasoning in large language models under practical post-training constraints. They found that while classification performance recovered strongly with self-generated response finetuning, generative reasoning capabilities consistently showed a persistent and substantial performance gap, linked to fundamental degradations in arithmetic computation and syntactic structure tracking.
Key insights
- Layer pruning, even with strong SGR-based recovery, creates a persistent and significant performance gap between classification tasks (which recover well) and generative reasoning tasks.
- The degradation in generative reasoning is not merely superficial text generation issues but stems from impaired core algorithmic capabilities, such as arithmetic computation and the ability to maintain syntactic structures.
- These fundamental algorithmic degradations prove difficult to fully restore, even with extensive post-pruning distillation or targeted, resource-intensive finetuning efforts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.