Reproduce “RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Byeongguk Jeon, Seonghyeon Ye, JaeHyeok Doo, Sungdong Kim, Minjoon Seo, Hyungmok Son, Kimin Lee
RoboWorld presents a neural simulation pipeline for evaluating generalist robot policies, integrating a fast autoregressive video world model with a task-progress-aware Vision-Language Model judge. This approach achieved a Pearson correlation of 0.989 and a Spearman correlation of 0.970 with real-world policy rankings, significantly reducing evaluation costs and time.
Key insights
- STEP FORCING, by training the world model to predict clean frames from 'one-step self-forwarded priors' under the same few-step denoising schedule, effectively addresses the train-test context mismatch and reduces error accumulation in long-horizon autoregressive generation.
- A nuanced, task-progress-aware VLM rubric combined with a multi-view evaluation strategy (using stable external views for task progress and wrist view for world-model errors) significantly enhances the reliability of policy scoring by distinguishing genuine policy failures from simulation artifacts.
- Despite inherent imperfections in world models, a carefully designed neural simulation system can achieve a high correlation with real-world robot policy rankings, offering a scalable and cost-effective alternative to physical evaluation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.