Reproduce “RAEE: A Robust Retrieval-Augmented Early Exit Framework for Efficient Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lianming Huang, Shangyu Wu, Yufei Cui, Ying Xiong, Haibo Hu, Xue Liu, Tei-Wei Kuo, Nan Guan and 1 more
RAEE, a retrieval-augmented early exit framework from researchers at City University of Hong Kong, Mohamed bin Zayed University of Artificial Intelligence, MILA, and National Taiwan University, improves large language model inference by dynamically selecting optimal exit layers based on similar data. It reduces latency by nearly half for billion-level models like Llama-3-8B and Gemma-7B while also enhancing predictive accuracy across various NLP tasks, sometimes surpassing the full model's performance.
Key insights
- Intermediate layers of large language models can sometimes make more accurate predictions than the final output, suggesting early exit can correct full model errors.
- Inputs with similar semantic content tend to have consistent optimal exit layers, making it feasible to infer exit behavior for new data from retrieved similar examples.
- The early exit problem can be effectively modeled as predicting a discrete probability distribution over potential exit layers, which can be approximated by aggregating information from retrieved neighbors.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.