Reproduce “Real-World Offline Reinforcement Learning from Vision Language Model Feedback” on OpenResearch

About this paper

Sreyas Venkataraman, Yufei Wang, Ziyu Wang, Navin Sriram Ravie, Zackory Erickson, David Held

Offline RL-VLM-F trains a robot to pull a garment sleeve toward a shoulder from recorded attempts. It uses a vision-language model to compare images with a task description, learns reward preferences, and applies offline RL to point-cloud data. On a ViperX arm, it achieved a dressed ratio of 0.83 versus 0.32 for DP3; both averaged 0.9 on a manikin with the garment grasped and limb static.

Key insights

• Vision-Language Models can effectively infer meaningful reward signals from high-level textual goals and visual observations, even when low-level state information is unavailable.
• The preference-based reward learning approach is robust to noise and occasional errors in VLM-generated labels, with performance gracefully degrading but still effective up to 25% random preference errors.
• Learning an explicit reward function through VLM feedback for offline RL leads to policies with enhanced generalization capabilities, particularly for out-of-distribution scenarios, compared to direct imitation learning.

Summary by alphaXiv.