Reproduce “Beyond Policy Optimization: A Data Curation Flywheel for Sparse-Reward Long-Horizon Planning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yutong Wang, Pengliang Ji, Kaixin Li, Baolong Bi, Tao Feng, Guillaume Sartoretti
A new data curation framework, BPO, trains Large Language Model agents for complex, sparse-reward, long-horizon planning by using terminal rewards as a data filter for self-improvement. The framework achieves state-of-the-art success rates on benchmarks like ALFWorld and ScienceWorld, outperforming larger models while significantly improving reasoning efficiency.
Key insights
- Sparse terminal rewards, while ineffective for gradient-based policy optimization in long-horizon tasks, serve as a highly effective and unambiguous signal for curating successful trajectories.
- An iterative, reward-gated data curation flywheel enables continuous self-improvement and robust learning for LLM agents in complex environments by focusing on successful experiences.
- Balancing deep reasoning with contextual efficiency is achievable through explicit distillation of verbose thoughts into concise planning thoughts, significantly reducing computational overhead without sacrificing performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.