Reproduce “LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mohammad Rastegari, Mahyar Najibi
Researchers from Apple and Meta AI developed LazyLLM, a training-free dynamic token pruning method that significantly accelerates long-context Large Language Model inference by selectively processing essential tokens. The approach achieves up to 4.77x speedup in time-to-first-token for models like XGen 7B on synthetic tasks with negligible accuracy degradation.
Key insights
- Prompt tokens exhibit significant redundancy, as attention scores for next-token prediction are often sparse, indicating that many tokens contribute minimally to immediate output.
- The relevance of tokens for prediction can evolve across different generation steps, necessitating a dynamic pruning mechanism that allows for re-inclusion of previously pruned tokens.
- Later transformer layers are less sensitive to token pruning compared to earlier layers, which informs an effective progressive pruning strategy for balancing efficiency and accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.