Reproduce “Embodied First Exam” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kangrui Wang, Qineng Wang, Changhe Chen, Pingyue Zhang, Chi Gui, Feishi Wang, Canyu Chen, Wenli Xiao and 22 more
The Embodied First Exam (EFE) evaluates coding agents controlling robots across direct control, controller-writing, and policy-training approaches. It standardizes tasks, information access, resources, and external verification across 1,156 tasks, 35 suites, and 47 robots. On 391 paired core tasks, GPT-6 Astra solved 30% in Standard and 77% in Privileged; these simulated results compare access conditions, not real-world readiness.
Key insights
- Access to resets improved performance more than exact object-state access alone in controlled comparisons, particularly for tasks requiring precise alignment or contact.
- Failures were often shared across agents at the same task stage, and difficult stages commonly involved material changes, dynamic interactions, or whole-body movement.
- Model choice was not the only determinant: harness changes shifted success rates by up to 12 percentage points, and stronger agents still solved partly different sets of tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.