Reproduce “ALPS: Attention Localization and Pruning Strategy for Efficient Alignment of Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hao Chen, Haoze Li, Zhiqing Xiao, Lirong Gao, Qi Zhang, Xiaomeng Hu, Ningtao Wang, Xing Fu and 1 more

Researchers from Zhejiang University and Ant Group developed ALPS, an attention localization and pruning strategy that identifies task-sensitive attention heads by analyzing weight parameter shifts. This method updates only 10% of attention parameters, leading to a 2.75% performance gain over full fine-tuning on a 1B Llama-3 model, approximately 5x faster training, and improved knowledge preservation across tasks.

Key insights

  • LLM model weight parameters inherently encode task-relevant information, enabling data-independent identification of task-sensitive components.
  • The Wasserstein-1 distance is a robust metric for quantifying subtle yet significant parameter shifts in attention heads, crucial for effective task alignment.
  • Fine-tuning only a small, task-sensitive subset of attention heads can yield superior performance and efficiency compared to updating all parameters or a random subset.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.