Reproduce “Deliberative Searcher: Improving LLM Reliability via Reinforcement Learning with constraints” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhenyun Yin, Shujie Wang, Xuhong Wang, Xingjun Ma, Yinchun Wang
Researchers at Fudan University and Shanghai AI Lab developed the Deliberative Searcher, an LLM framework that integrates continuous confidence calibration into search-augmented reasoning. It utilizes constrained reinforcement learning to significantly reduce "false & certain" errors, achieving up to a 96% reduction, and improves test-time inference efficiency by up to 4x through confidence-weighted aggregation.
Key insights
- Achieving well-calibrated confidence in LLMs requires explicit optimization via constrained reinforcement learning, rather than merely relying on model scale or retrieval quality.
- The adaptive Lagrangian formulation within the RL framework effectively balances correctness and reliability by dynamically penalizing miscalibration.
- Calibrated confidence enables substantial computational efficiency improvements during inference through confidence-weighted aggregation, outperforming traditional majority voting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.