Reproduce “Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lang Feng, Weihao Tan, Zhiyi Lyu, Longtao Zheng, Haiyang Xu, Ming Yan, Fei Huang, Bo An
This research introduces Counterfactual Soft Reinforcement Learning (CoSo), a method for efficient online fine-tuning of Vision-Language Model agents. CoSo leverages counterfactual reasoning to identify the causal influence of individual tokens on the final executable action, leading to more targeted exploration and outperforming existing online RL fine-tuning methods by 9-16% across diverse VLM tasks like Android control, card gaming, and embodied AI.
Key insights
- Not all tokens within a VLM's textual output (utterance) equally influence the final executable action, with less than 10% typically being action-critical.
- Traditional reinforcement learning exploration methods, like naive entropy regularization, are inefficient in open-ended textual action spaces because they treat all tokens uniformly, wasting exploration on semantically redundant elements.
- Quantifying the causal influence of individual tokens on the parsed action enables a more targeted and efficient online exploration strategy, leading to faster policy optimization and higher task success.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.