Reproduce “Self-Guided Test-Time Training for Long-Context LLMs” on OpenResearch

About this paper

Xinyu Zhu, Zhe Xu, Xiaohan Wei, Yunchen Pu, Fei Tian, Chonglin Sun, Kaushik Rangadurai, Hua Zhi and 5 more

Researchers from Meta AI and the University of Virginia introduced Self-Guided Test-Time Training (S-TTT), a method where large language models select their own question-relevant context spans for adaptation, yielding consistent performance improvements, including up to a 15% relative gain in accuracy for long-context reasoning tasks, by optimizing the quality of test-time training data. This approach consistently outperformed existing test-time training baselines on LongBench-v2 and LongBench-Pro benchmarks.

Key insights

• The effectiveness of test-time training for long-context LLMs is critically dependent on the quality and relevance of the training tokens, as demonstrated by the superior performance with "oracle" spans.
• Leveraging the LLM's own question-conditioned relevance judgment to select evidence spans is more effective than relying on intrinsic metrics like perplexity or entropy for curating test-time training data.
• S-TTT induces targeted attention shifts in the middle layers of the LLM, strengthening focus on the trained spans without broadly altering attention patterns across the entire context.

Summary by alphaXiv.