Reproduce “CAFE: Retrieval Head-based Coarse-to-Fine Information Seeking to Enhance Multi-Document QA Capability” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Han Peng, Jinhao Jiang, Zican Dong, Wayne Xin Zhao, Lei Fang
The CAFE framework introduces a training-free, two-stage approach that leverages specific retrieval heads within large language models to refine information from long multi-document contexts for question answering. It achieves substantial performance gains on MDQA tasks by progressively filtering irrelevant documents and steering attention towards critical evidence, thereby mitigating the 'lost-in-the-middle' problem.
Key insights
- Specific 'retrieval heads' within LLMs can be identified and strategically utilized to gauge document relevance to a given query, forming the basis for an internal information-seeking mechanism.
- Mimicking human cognitive processes, a two-stage coarse-to-fine approach effectively manages long-context inputs by progressively filtering irrelevant documents and then focusing attention on critical evidence.
- Locality-based re-ranking, which strategically positions relevant documents later in the input, combined with inference-time attention steering, proves effective in counteracting the 'lost-in-the-middle' phenomenon and the negative impact of distracting information.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.