Reproduce “ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, Dieter Fox
ALFRED is a new benchmark for embodied AI that challenges agents to interpret natural language instructions and egocentric vision to complete complex, multi-step household tasks in a simulated 3D environment. Baseline models achieved very low task success rates (e.g., 2.1% seen, 0.5% unseen), significantly below human performance (91.0%), highlighting the benchmark's difficulty in requiring long-horizon planning and fine-grained object manipulation.
Key insights
- The benchmark demonstrates a significant gap for current AI models, with human agents achieving 91.0% task success while strong baselines reach only 0.4%-4.0% on complex tasks.
- Successful task completion requires robust multimodal reasoning, as both visual perception and language understanding are indispensable, with unimodal models achieving 0% task success.
- Models struggle with generalizing learned policies and object interactions to novel scenes and object instances, as performance consistently drops on "unseen" environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.