Reproduce “RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Huang Huang, Wensi Ai, Ziyu Chen, Youhui Wang, Zijian Du, Yang Liu, Jiaolong Yang, Li Fei-Fei and 1 more
RoboRender turns simulated robot trajectories into realistic RGB video for training policies, using depth to preserve geometry, robot masks to retain motion cues, and language prompts to vary appearance while keeping simulator-provided states and actions. Across eight tested real-robot tasks, its policy averaged 71% success, versus 20% and 10% for baselines; results are limited to these tasks and setups.
Key insights
- Conditioning video generation on depth and robot masks ties visual appearance to simulated geometry and robot motion, rather than generating unconstrained robot footage.
- Joint multi-view generation is designed to preserve consistency across cameras; generating views independently or merely stacking them does not provide comparable consistency in the reported tests.
- Visual variation can improve policy learning without adding new simulated motion trajectories: in reduced-data experiments, additional generated versions improved success before gains saturated.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.