Reproduce “SeeUPO: Sequence-Level Agentic-RL with Convergence Guarantees” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianyi Hu, Qingxu Fu, Yanxi Chen, Zhaoyang Liu, Bolin Ding
A reinforcement learning algorithm called SeeUPO, developed by Tongyi Lab at Alibaba Group, offers convergence guarantees for training large language model (LLM) agents in multi-turn interactions. It delivers up to 54.6% relative performance improvements over baselines with stable training, operating as a critic-free method.
Key insights
- Mainstream RL algorithms fail to simultaneously achieve critic-free operation and robust convergence guarantees in multi-turn settings due to inherent biases in critic-free advantage estimation or the complexity of critic learning.
- Modeling multi-turn interactions as a multi-agent contextual bandit problem with reverse-order policy updates (backward induction) enables convergence to the globally optimal policy.
- Batch-level advantage normalization is theoretically proven to preserve convergence guarantees while improving numerical stability, unlike some group-level normalizations that can break them.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.