Reproduce “RoboColosseum: Evaluating Physical Intelligence at Scale In-The-Wild” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tan Sang Nguyen, Yi Ru Wang, Shailesh Shailesh, Gang Yang, Jiamo Hu, Jianuo Jiang, Haoquan Fang, Zhongzheng Ren and 19 more
RoboColosseum evaluates robot policies across locally operated sites using standardized protocols and two tracks: blinded, back-to-back comparisons and fine-tuning on predefined tasks with a shared demonstration budget. It also measures partial task progress alongside completion. In five G1 tasks, none of five policies succeeded, but mean normalized progress ranged from 13.3% to 23.2%; this finding is specific to that suite.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.