Reproduce “Goal-VLA: Image-Generative VLMs as Object-Centric World Models Empowering Zero-shot Robot Manipulation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Haonan Chen, Jingxiang Guo, Bangjun Wang, Tianrui Zhang, Xuchuan Huang, Boren Zheng, Yiwen Hou, Chenrui Tie and 2 more

Researchers from the National University of Singapore developed Goal-VLA, a hierarchical framework for zero-shot robot manipulation that utilizes image-generative Vision-Language Models as an object-centric world model. The system translates visual goal states into precise 3D object poses, achieving an average 59.9% success rate on RLBench tasks and 60% in real-world experiments without task-specific training.

Key insights

  • Decoupling high-level semantic goal reasoning (via VLM-generated images) from precise low-level spatial grounding is key to achieving robust zero-shot generalization in robotic manipulation.
  • An object-centric visual goal state, derived from an image-generative VLM, can serve as a potent interface, providing both semantic understanding and explicit spatial guidance for robot control.
  • An iterative self-correction mechanism, termed "Reflection-through-Synthesis," significantly enhances the robustness and accuracy of generated visual goals by allowing the system to refine potentially infeasible or incorrect outcomes.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.