Reproduce “Agentic Policy Optimization via Instruction-Policy Co-Evolution” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Han Zhou, Xingchen Wan, Ivan Vulić, Anna Korhonen
The INSPO framework enables Large Language Model (LLM) agents to dynamically co-evolve their guiding instructions with their policies during Reinforcement Learning (RL), moving beyond static prompt engineering. This approach improves LLM agent performance in complex multi-turn reasoning tasks, achieving an average 6% Exact Match (EM) score increase over state-of-the-art baselines for the Qwen-2.5-3B model.
Key insights
- Optimal instructions for LLM agents are dynamic and require online co-evolution with the agent's policy, rather than being static or optimized in an offline manner.
- An on-policy reflection mechanism that leverages failure trajectories and an LLM-based optimizer to critique and refine instructions is effective for generating strategic and adaptable guidance.
- Maintaining a dynamic population of instructions with pruning, expansion, and verification steps is crucial for sustaining performance improvements and guiding effective policy learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.