Reproduce “Investigating Layer Importance in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yang Zhang, Yanfei Dong, Kenji Kawaguchi
Researchers at the National University of Singapore employed Shapley values and layer ablation to identify "cornerstone layers" in Large Language Models, demonstrating that a few early layers disproportionately contribute to overall performance, accounting for 29% to 37% of total model output. Removing these cornerstone layers critically collapses model functionality, whereas removing other layers has minimal impact.
Key insights
- A small subset of "cornerstone layers," consistently found in the early stages of LLMs, disproportionately contributes to overall model performance.
- This concentration of importance becomes more pronounced in larger LLM architectures (e.g., LLaMA3-70B vs. LLaMA3-8B).
- Non-cornerstone layers exhibit marginal individual contributions, suggesting a more distributed or potentially overlapping functional role, allowing other layers to compensate for their absence.
- Mixture-of-Experts (MoE) models, like Mixtral-8x7B, show greater robustness to cornerstone layer ablation, indicating a less singular reliance on individual layers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.