Reproduce “Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tai An, Ruwu Cai, Yanzhe Zhang, Yang Liu, Hao Chen, Pengcheng Xie, Sheng Chang, Yiwu Yao and 1 more

Researchers from Huawei Technologies Co., Ltd. introduce Amber Pruner, a training-free algorithm that applies N:M activation sparsity to accelerate the prefill stage of Large Language Models (LLMs). This method, integrated with W8A8 quantization in the Outstanding-sparse framework, achieves over 55% acceleration coverage of linear computations with less than 1% average accuracy loss on zero-shot tasks for 8:16 sparsity across diverse LLM architectures.

Key insights

  • LLM activation values exhibit a higher proportion of elements near zero compared to weights, making them inherently more suitable for structured N:M sparsification without extensive retraining.
  • Combining a weight-aware scoring function with a selective layer-skipping strategy is crucial for achieving N:M activation sparsity while preserving model accuracy across various tasks.
  • The interaction between quantization and sparsity can be synergistic; modified quantization scaling can reshape activation distributions to explicitly expose more inherent sparsity, improving pruning effectiveness.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.