Reproduce “VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kuan-Chen Chen, Winston Chen, Wei-Fang Sun, Min-Chun Hu
VLM-AR3L presents a framework that leverages Vision-Language Models (VLMs) to create both absolute and relative reward signals for reinforcement learning, providing stable and robust supervision. This dual-reward approach addresses the limitations of single-reward formulations in complex visual environments and allows RL agents to learn policies without manual reward engineering, matching or exceeding human-designed reward performance in diverse tasks.
Key insights
- Modern VLMs, such as Gemini-2.0-Flash, reliably generate high-accuracy preference labels for complex visual observations and language goals, outperforming similarity-based models.
- Explicitly training a dedicated Siamese network for relative rewards is crucial for stable comparison and computational efficiency, surpassing direct VLM querying or absolute reward differences.
- The combination of absolute and relative reward signals provides complementary benefits, particularly with relative rewards proving effective in tasks with ambiguous state ordering or cyclic structures where absolute rewards are less robust.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.