Reproduce “Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Weizhi Fei, Xueyan Niu, Guoqing Xie, Yingqing Liu, Bo Bai, Wei Han
Researchers from Tsinghua University and Huawei Technologies developed EHPC, a training-free prompt compression method that identifies and utilizes specialized "evaluator heads" within existing large language models to efficiently reduce input token length. This approach achieved state-of-the-art performance on LongBench and ZeroSCROLLS, notably reducing compression latency to 0.88 seconds for a 14,354-token prompt while improving accuracy on question answering tasks.
Key insights
- Specific attention heads, termed 'evaluator heads', exist within LLM transformer architectures that are inherently adept at identifying critical information in long input contexts.
- These evaluator heads are largely task-agnostic and robust, meaning they can be identified once using simple data and then applied effectively across various downstream tasks.
- Leveraging these intrinsic attention mechanisms for token deletion during the pre-filling stage offers an efficient, training-free, and high-performing prompt compression strategy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.