Reproduce “Learning Visual Parkour from Generated Images” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alan Yu, Ge Yang, Ran Choi, Yajvan Ravan, John Leonard, Phillip Isola

A framework called LucidSim trains robot policies for visual parkour using only images generated by a combination of physics simulation and generative AI. This approach enables zero-shot transfer of RGB-only locomotion skills to a physical robot, outperforming traditional domain randomization methods.

Key insights

  • Generative AI, guided by physics simulations, can produce diverse and physically consistent visual data for robot learning, effectively addressing the visual reality gap.
  • On-policy learning via DAgger iterations is critical for a visual policy to achieve robust performance, surpassing the benefits of merely scaling expert data.
  • RGB-only policies, trained on rich generated data, can generalize better than depth-only policies, particularly in dynamic environments with varied backgrounds.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.