Reproduce “Rethinking with Retrieval: Faithful Large Language Model Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hangfeng He, Hongming Zhang, Dan Roth
This paper introduces 'Rethinking with Retrieval' (RR), a post-processing method that enhances the factual faithfulness and performance of Large Language Models (LLMs) on complex reasoning tasks. The approach leverages intermediate reasoning steps generated by LLMs to guide targeted external knowledge retrieval, outperforming Chain-of-Thought baselines and improving accuracy on tasks like StrategyQA and TempQuestions without requiring LLM fine-tuning.
Key insights
- Decomposing LLM-generated reasoning into individual steps for targeted knowledge retrieval is more effective than retrieving based on the original query alone, significantly improving performance.
- A lightweight, post-processing approach can effectively integrate external knowledge to enhance both predictive accuracy and the factual faithfulness of LLM explanations without requiring expensive training or fine-tuning.
- The benefits of 'Rethinking with Retrieval' are consistent across various LLM sizes, demonstrating its broad applicability from smaller OPT models to GPT-3.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.