Reproduce “OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Seungjun Shin, Jaehoon Oh, Dokwan Oh
OrthoRank introduces a training-free token selection method for efficient Large Language Model inference by leveraging the observed orthogonality between non-sink tokens and the static attention sink token. This approach dynamically prunes computation, achieving superior performance over layer pruning methods, exemplified by a 1.18x throughput improvement for Llama-2-13B at 20% sparsity with better perplexity and accuracy.
Key insights
- Beyond attention, other tokens' normalized hidden states gradually align with the attention sink token in deeper layers, while the sink token itself remains almost static.
- Tokens more orthogonal to the static attention sink token exhibit a greater potential for change, signifying their importance for model updates.
- Maintaining Key and Value (KV) calculations for all tokens, including those not selected for full computation, is crucial for preserving contextual information and preventing significant performance degradation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.