Reproduce “The Great March 100: 100 Detail-oriented Tasks for Evaluating Embodied AI Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziyu Wang, Chenyuan Liu, Yushun Xiang, Runhao Zhang, Qingbo Hao, Hongliang Lu, Houyu Chen, Zhizhong Feng and 11 more
The Great March 100 (GM-100) benchmark provides a collection of 100 systematically designed, detail-oriented tasks for evaluating embodied AI agents, addressing limitations in current evaluation approaches. Experiments demonstrate that existing Vision-Language-Action models achieve low success rates on these challenging tasks, though a strong inverse correlation is observed between action prediction error and physical task completion.
Key insights
- Current robot learning benchmarks overemphasize common actions, resulting in models with significant biases and limited generalization capabilities.
- A principled approach to task design, integrating human-object interaction knowledge and low-level manipulation understanding, is essential for creating truly challenging and comprehensive benchmarks.
- High-precision action modeling is a prerequisite for robust physical interaction in complex tasks, as evidenced by a strong inverse correlation between action prediction error and real-world task success.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.