Reproduce “Self-Taught Agentic Long Context Understanding” on OpenResearch

About this paper

Yufan Zhuang, Xiaodong Yu, Jialian Wu, Ximeng Sun, Ze Wang, Jiang Liu, Yusheng Su, Jingbo Shang and 2 more

Researchers from AMD and UC San Diego developed AgenticLU, a framework that enhances large language models' ability to understand long contexts by teaching them to self-clarify and ground information. This system distills complex multi-step reasoning into a single-pass inference model, achieving 71.1% accuracy on HotpotQA with 128K tokens, which is a substantial improvement over Llama3.1-8B's 40.0% and ProLong-8B's 58.9%.

Key insights

• LLMs can effectively "teach themselves" sophisticated reasoning and retrieval strategies for long-context understanding by generating and distilling their own iterative clarification paths.
• Explicit self-clarification and contextual "pointback" mechanisms are crucial for bridging the gap between nominal and effective context windows, directly addressing the "lost-in-the-middle" problem.
• The computational cost of complex, multi-step agentic reasoning can be amortized from inference time to training time through finetuning, resulting in minimal runtime overhead (1.93% increase) for enhanced long-context capabilities.

Summary by alphaXiv.