Reproduce “ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yu Li, Rui Miao, Zhengling Qi, Tian Lan
ARISE introduces a hierarchical reinforcement learning framework for large language models, enabling the intrinsic evolution of a skill library to enhance mathematical reasoning. The method consistently improved performance across diverse math benchmarks, achieving an average 2.9 percentage point gain on out-of-distribution Omni-MATH problems by integrating skill selection and generation directly into the policy optimization.
Key insights
- Directly integrating skill management within the policy via log-probabilities allows the hierarchical advantage signal to effectively guide skill acquisition and application, overcoming limitations of external retrievers.
- A hierarchical reward system, providing a bonus for correct solutions achieved with skill utilization, is crucial for driving skill adoption and effectively co-evolving the policy and its skill library.
- Dynamically growing and curating a two-tier skill library (cache and reservoir) within the RL framework is essential for improving reasoning performance, particularly for out-of-distribution generalization, by accumulating and efficiently using reusable reasoning patterns.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.