Reproduce “GoalLadder: Incremental Goal Discovery with Vision-Language Models” on OpenResearch

About this paper

Alexey Zakharov, Shimon Whiteson

A method called GoalLadder, developed by researchers at the University of Oxford, enables reinforcement learning agents to learn visual tasks from a single natural language instruction by incrementally discovering goals using Vision-Language Models. It mitigates noisy VLM feedback with an ELO-based rating system and improves VLM query efficiency through a learned visual embedding space, achieving approximately 95% success rates across continuous control tasks with about 4500 VLM queries for Metaworld.

Key insights

• An ELO-based rating system can effectively absorb and aggregate noisy feedback from Vision-Language Models, leading to a robust ranking of task sub-goals.
• Defining reward functions as distances in a learned, compact visual embedding space (VAE) allows for efficient reward signal generalization to unseen states, significantly reducing the need for continuous VLM queries.
• An incremental discovery process, guided by a VLM, can progressively identify and refine task-relevant sub-goals from a single natural language instruction.

Summary by alphaXiv.