Reproduce “Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shuo He, Lang Feng, Qi Wei, Xin Cheng, Lei Feng, Bo An
Hierarchy-of-Groups Policy Optimization (HGPO), developed by researchers at Nanyang Technological University and Southeast University, introduces a reinforcement learning algorithm that resolves historical context inconsistency in stepwise group-based training for long-horizon agentic tasks. It achieved state-of-the-art performance with a 94.85% in-distribution success rate on ALFWorld and improved generalization capabilities by refining advantage estimation through context-aware hierarchical grouping.
Key insights
- Historical context inconsistency is a fundamental source of bias in advantage estimation for stepwise group-based RL in multi-turn LLM agent tasks.
- A bias-variance trade-off exists: more consistent historical contexts yield lower-bias advantage estimates but are less frequent, while less consistent contexts are abundant but introduce higher bias.
- By systematically aggregating advantage estimates across different levels of context consistency, HGPO effectively interpolates this trade-off, providing a more robust and accurate training signal.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.