Reproduce “Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell” on OpenResearch

About this paper

Taiming Lu, Muhan Gao, Kuai Yu, Adam Byerly, Daniel Khashabi

Johns Hopkins University researchers identified a "Know but Don't Tell" phenomenon in large language models (LLMs) when processing long contexts. Their probing analysis revealed that LLMs consistently encode the location of crucial information internally with near-perfect accuracy, yet often fail to leverage this knowledge for accurate answer generation, particularly for information located in the middle of the input.

Key insights

• LLMs exhibit a "Know but Don't Tell" phenomenon, where they accurately encode the position of target information internally but often fail to incorporate this knowledge into their generated responses.
• Information located in the middle of a long context requires more processing layers for accurate internal localization by LLMs compared to information at the beginning or end.
• A strong negative correlation exists between how deep into the network an LLM achieves peak positional understanding and its final answer generation accuracy, indicating that delayed internal localization contributes to failures.
• Positional biases influencing the model's final output probability (logits) primarily emerge in the later layers of the transformer architecture.
• Even when successfully localized, information from the middle of the context can become less accessible or more entangled in very late layers, impacting subsequent retrieval for generation.

Summary by alphaXiv.