Reproduce “LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yeqin Zhang, Yunfei Wang, Jiaxuan Chen, Ke Qin, Yizheng Zhao, Cam-Tu Nguyen
Researchers introduce methods to create sentence embeddings from Large Language Models by aggregating attention value vectors, demonstrating these vectors encode sentence semantics more effectively than traditional hidden states. The proposed techniques achieve state-of-the-art performance among training-free methods with improved computational efficiency.
Key insights
- Attention value vectors within LLM Transformers are better suited for encoding global sentence-level semantics compared to hidden states, which prioritize next-token prediction.
- Sentence meaning can be conceptually linked to a sentence's influence on its continuation space, and value vectors serve as more effective proxies for this influence.
- Aligning aggregated value vectors with the LLM's residual stream space via the output projection matrix significantly enhances embedding quality for downstream tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.