Reproduce “Making Long-Context Language Models Better Multi-Hop Reasoners” on OpenResearch

About this paper

Yanyang Li, Shuo Liang, Michael R. Lyu, Liwei Wang

A novel approach called Reasoning with Attributions enhances multi-hop reasoning in long-context language models by requiring explicit citations during the reasoning process. This method allows smaller open-source models, like a fine-tuned Vicuna-7B, to achieve competitive performance against proprietary models such as ChatGPT and Claude-instant, while also improving robustness to noisy contextual information.

Key insights

• Explicit attribution (Chain-of-Citation) consistently improves multi-hop reasoning performance and robustness compared to standard Chain-of-Thought prompting, guiding the model to ground its claims in the provided context.
• Fine-tuning smaller open-source long-context LMs with a specialized attribution-guided dataset and multi-task learning can enable them to match or surpass the multi-hop reasoning capabilities of larger, proprietary models.
• A strong positive correlation exists between citation precision and multi-hop reasoning performance, suggesting that attribution quality is a reliable indicator of the model's overall reasoning effectiveness.

Summary by alphaXiv.