Reproduce “Pruning Large Language Models with Semi-Structural Adaptive Sparse Training” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Weiyu Huang, Yuezhou Hu, Guohao Jian, Jun Zhu, Jianfei Chen

Researchers at Tsinghua University developed Adaptive Sparse Trainer (AST), an efficient retraining framework that produces 2:4 semi-structured sparse Large Language Models (LLMs) with performance competitive to dense models, achieving up to 1.83x inference speedup on specific hardware with minimal retraining cost.

Key insights

  • Dynamically learning optimal sparse masks with an annealing decay schedule (ARS-STE) allows for a smooth transition from dense to sparse and effective exploration of connectivity patterns during retraining.
  • Knowledge distillation from a dense teacher model is critical for guiding sparse LLMs to better global optima and preserving complex abilities, overcoming the "retraining dilemma" of pre-trained sparse models.
  • Initializing LoRA-like adapters with information from pruned weights (SLoRB) and co-training them with sparse parameters effectively compensates for reduced model capacity, particularly under limited retraining budgets.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.