Reproduce “Transferring the Intelligence of VLMs to Robotic Control” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Meng-Hao Guo, Zhe-Han Mo, Jia-Jun Wang, Yi Zhang, Kejin Wang, Yi-Xuan Deng, Jia-Peng Zhang, Yongming Rao and 1 more

RoboDawn lets a frozen vision-language model control robots through move, rotate, and grip commands, using a shared primer, optional task demonstrations, and feedback from execution. On RoboTwin 2.0 C2R, GPT-6 Astra reaches 53.2% average task success with zero task demonstrations and 73.6% with one, versus 46.0% for trained π0.5. This supports transfer on tested manipulation benchmarks, not general embodied intelligence.

Key insights

  • Interface design can make some pretrained VLM capabilities usable for closed-loop manipulation without fine-tuning, though performance depends strongly on the selected model.
  • One task demonstration often provides the largest improvement over zero-shot control; additional examples have diminishing returns and can hurt when they make the context too long.
  • The approach is better suited to higher-level manipulation than rapid or precision-critical control: discrete action increments, latency, and execution errors remain constraints.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.