Reproduce “Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Training” on OpenResearch

About this paper

Junqing He, Kunhao Pan, Xiaoqun Dong, Zhuoyang Song, Yibo Liu, Qianguo Sun, Yuxin Liang, Hao Wang and 2 more

Researchers at the International Digital Economy Academy (IDEA), Shenzhen, introduced Position-Agnostic Decompositional Training (PAM QA) to address the "lost in the middle" problem in LLMs. Their Ziya2-Reader model achieved 99% accuracy in retrieving information regardless of its position within long contexts, and obtained a 7.0% absolute gain in Rouge-L score for Chinese Multi-document QA over prior state-of-the-art models.

Key insights

• The "lost in the middle" problem is primarily an attention distribution failure, where LLMs neglect middle or tail tokens due to decaying attention scores, rather than a lack of context window capacity.
• Explicitly training LLMs to perform multi-step reasoning, including question re-statement and index prediction, forces them to distribute attention more evenly and effectively search for information regardless of its position.
• Position-agnostic data generation, involving random shuffling of documents and hard negative sampling, is crucial for preventing positional biases and enhancing the model's ability to discriminate relevant information from noise.

Summary by alphaXiv.