Reproduce “EMMOE: A Comprehensive Benchmark for Embodied Mobile Manipulation in Open Environments” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dongping Li, Tielong Cai, Tianci Tang, Wenhao Chai, Katherine Rose Driggs-Campbell, Gaoang Wang
Researchers from Zhejiang University, UIUC, and the University of Washington developed EMMOE, a comprehensive benchmark for evaluating embodied mobile manipulation using natural language instructions in open environments. Their HOMIEBOT agent system, trained on the EMMOE-100 dataset, demonstrates strong performance across novel metrics like Task Progress and Success End Rate.
Key insights
- Physical grounding failures (F2 errors), where models output non-existent or unrecognized targets, are the predominant obstructive factor in failed embodied tasks, indicating a core challenge for LLMs in real-world environments.
- Direct Preference Optimization (DPO), while effective for in-domain training, may compromise generalization and transferability to unseen tasks in complex, multi-stage robotic domains.
- The 'Pick' action represents a significant bottleneck in low-level manipulation, exhibiting lower success rates and higher execution errors compared to other actions, pinpointing an area for future improvement in robotic skill learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.