Reproduce “Discover, Learn, and Reinforce: Scaling Vision-Language-Action Pretraining with Diverse RL-Generated Trajectories” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rushuai Yang, Zhiyuan Feng, Tianxiang Zhang, Kaixin Wang, Chuheng Zhang, Li Zhao, Xiu Su, Yi Chen and 1 more

The DLR (Discover, Learn, and Reinforce) framework scales Vision-Language-Action pre-training by algorithmically generating diverse, high-quality robotic trajectories using multi-pattern reinforcement learning. This approach improves VLA model generalization and demonstrates positive data-scaling behavior during downstream adaptation, addressing limitations of human teleoperation and standard RL.

Key insights

  • Decoupling the objectives of diversity and task success is critical; DLR seeks diversity only among trajectories that have already achieved task success, resolving the exploration conflict.
  • Online reinforcement learning can effectively refine multiple distinct, pattern-conditioned policies without causing mode collapse, provided they are initialized within separate successful behavioral manifolds.
  • Leveraging offline human demonstrations for initial pattern discovery, then using online RL for scalable generation, combines the strengths of both data acquisition methods.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.