Reproduce “Dense Process Supervision for Search Agents via Fact Utility Estimation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rongzhi Zhu, Xiangyu Liu, Yi Liu, Shuo Zhang, Ruirui Zhang, Rui Wu, Tao Jiang, Zequn Sun and 2 more
FactAgent, a reinforcement learning framework for large language model search agents, enhances training by explicitly representing gathered information as structured facts and estimating their utility to provide dense, step-level rewards. This approach yields improved performance, especially in multi-hop question answering tasks, and maintains a compact agent state.
Key insights
- Explicitly representing knowledge as structured facts provides a more efficient and tractable state for LLM agents, alleviating context window limitations and computational overhead.
- Dense process supervision, derived from the estimated utility of acquired facts, offers fine-grained, immediate feedback that effectively resolves the credit assignment problem for intermediate actions.
- Semantic clustering of facts and robust Bayesian utility estimation enable stable reward signal generation even with limited data, reinforcing the learning of valuable information gathering strategies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.