Reproduce “Reinforcing Language Agents via Policy Optimization with Action Decomposition” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Muning Wen, Ziyu Wan, Weinan Zhang, Jun Wang, Ying Wen
Researchers from Shanghai Jiao Tong University and University College London introduce Policy Optimization with Action Decomposition (POAD), a framework that resolves theoretical inconsistencies in token-level reinforcement learning for language agents. POAD achieved competitive performance in restricted action spaces and significantly outperformed baselines in unrestricted coding tasks, notably achieving ROC AUC scores comparable to GPT-4 with CAAFE using a smaller LLM.
Key insights
- Naive token-level optimization introduces a theoretical inconsistency where later tokens are disproportionately weighted, as validated by the discrepancy increasing with more intra-action tokens and decreasing γ_w.
- Setting the intra-action discount factor (γ_w) to 1.0 within a modified Bellman backup (BAD) resolves this inconsistency, aligning token-level optimization with action-level utility.
- This approach transforms the optimization complexity from exponential O(|V|^|a|) to linear O(|a| * |V|), making reinforcement learning feasible for language agents in unrestricted action spaces.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.