Reproduce “ThinkSum: Probabilistic reasoning over sets using large language models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Batu Ozturkler, Nikolay Malkin, Zhen Wang, Nebojsa Jojic
ThinkSum enhances large language model reasoning on complex tasks by decoupling the LLM's associative knowledge retrieval from external, structured probabilistic inference, achieving state-of-the-art performance on 13 challenging BIG-bench tasks and improved robustness. The method effectively imbues LLMs with 'slow thinking' capabilities, often exceeding human performance and doing so with smaller model variants.
Key insights
- Externalizing complex probabilistic inference allows LLMs to overcome their inherent brittleness and achieve robust, multi-step reasoning that is less sensitive to prompt design.
- The modular 'Think' and 'Sum' architecture effectively bridges the gap between LLM's associative strengths and the demands of deliberate, 'slow thinking' tasks, drawing inspiration from human System 1/System 2 cognition.
- Enhanced reasoning capabilities can be achieved even with significantly smaller LLM variants by offloading the complex aggregation to an efficient external inference engine, making advanced AI more accessible.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.