Reproduce “Probabilistic Tree-of-thought Reasoning for Answering Knowledge-intensive Complex Questions” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shulin Cao, Jiajie Zhang, Jiaxin Shi, Xin Lv, Zijun Yao, Qi Tian, Juanzi Li, Lei Hou

Researchers from Tsinghua University and Huawei Technologies developed Probabilistic Tree-of-Thought (ProbTree) reasoning, a method that decomposes complex questions into a hierarchical tree structure and uses a confidence-aware mechanism to integrate parametric knowledge, external retrieval, and sub-question reasoning. The approach consistently improves accuracy on knowledge-intensive question answering datasets, achieving gains of up to 8.0% F1 over prior state-of-the-art methods by mitigating negative retrieval and error propagation.

Key insights

  • A hierarchical tree structure for question decomposition allows LLMs to have a 'broader sight' and recover from local errors, unlike linear chain-of-thought methods.
  • Quantifying confidence using explanation logits enables intelligent selection between parametric knowledge, external retrieved knowledge, and aggregated reasoning from sub-questions, mitigating issues like negative retrieval.
  • Aggregating answers from child nodes allows for meta-reasoning, providing a robust mechanism to synthesize complex information and improve overall accuracy.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.