Reproduce “RL with Learnable Textual Feedback: A Bilevel Approach” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Utsav Singh, Sidhaarth Sredharan, Souradip Chakraborty, Amrit Singh Bedi
Researchers from the University of Central Florida, Carnegie Mellon University, and the University of Maryland introduce Bilevel Natural Language Actor-Critic (Bi-NAC), a framework that employs bilevel optimization to co-train large language models (LLMs) as an actor and a feedback-generating critic. This method generates actionable textual feedback, leading to improved reasoning performance, superior sample and parameter efficiency, and reduced training costs across various benchmarks compared to prior reinforcement learning approaches.
Key insights
- Explicitly aligning the feedback generator's objective with the actor's downstream performance through a bilevel optimization framework is crucial for creating truly 'actionable' textual feedback.
- Coordinated actor-critic training within a bilevel setup effectively mitigates the 'sparsity bottleneck' and 'advantage collapse' issues common in traditional RLVR, leading to more stable and efficient learning.
- A single LLM can effectively perform both feedback generation and response refinement within the Bi-NAC framework, offering computational efficiencies without compromising performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.