Reproduce “Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rahaf Abu Hara, Vaibbhav Murarri, Claudio Zito
Reflective Prompted Policy Optimization (R2PO) introduces a framework for LLM-based policy search that leverages detailed trajectory-level behavioral evidence to enable more precise policy revisions. Using a 20B open-weight LLM, R2PO achieved the highest mean best reward across 10 diverse environments, demonstrating faster learning and improved stability compared to prior LLM-based and some deep RL methods.
Key insights
- Providing trajectory-level behavioral evidence, beyond scalar rewards, enables LLMs to perform precise diagnostic reasoning and generate more effective policy revisions.
- A critical failure mode termed "salience bias" emerges when LLMs over-focus on vivid worst-case examples in multi-evidence reasoning, which R2PO's calibrated evidence package effectively mitigates.
- Separating the global policy proposal (Search-LLM) from local, behavior-grounded revision (Critic-LLM) within a two-stage framework enhances the efficiency and stability of LLM-based policy optimization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.