Reproduce “Pause-Tuning for Long-Context Comprehension: A Lightweight Approach to LLM Attention Recalibration” on OpenResearch

About this paper

James Begin, Namit Agrawal, Eshan Singh, Yicheng Fu, Sean O'Brien, Vasu Sharma, Kevin Zhu

Algoverse AI Research introduced "pause-tuning," a lightweight method to mitigate the "Lost-in-the-Middle" problem in Large Language Models by combining strategic pause token insertion with targeted fine-tuning. This approach improved information retrieval accuracy by up to 16.10% for long contexts in models like LLaMA 3.1 8B Instruct, enhancing their ability to process lengthy input sequences.

Key insights

• Pause tokens function as "attention anchors" that periodically prompt the model to re-engage with all input sections, counteracting attention decay over long sequences.
• The full pause-tuning approach, which combines pause token insertion with targeted fine-tuning, achieves a synergistic effect that significantly outperforms either component alone.
• Pause-tuning is a resource-efficient technique that can be applied to existing transformer-based LLMs without architectural modifications, making it widely compatible.

Summary by alphaXiv.