Reproduce “OPEx: A Component-Wise Analysis of LLM-Centric Agents in Embodied Instruction Following” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haochen Shi, Zhiyuan Sun, Xingdi Yuan, Marc-Alexandre Côté, Bang Liu
Researchers from Mila, Université de Montréal, and Microsoft Research introduced OPEx, a component-wise analysis framework for LLM-centric agents in embodied instruction following. This framework demonstrates substantial performance gains on the ALFRED benchmark, achieving 17.74% and 16.78% absolute improvements in Success Rate on test seen and unseen splits respectively, while operating with reduced data requirements and highlighting perception and low-level action as key bottlenecks.
Key insights
- LLM-centric designs can achieve strong performance in EIF tasks with significantly less in-domain data compared to traditional supervised learning approaches, leveraging LLMs' inherent reasoning capabilities.
- Visual perception models and low-level action execution capabilities are critical bottlenecks, acting as major limiting factors for current LLM-centric embodied agents.
- A multi-agent dialogue strategy, combined with the integration of explicit prior knowledge, substantially enhances reasoning, grounding, and exception handling, especially when perception and low-level action complexities are minimized.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.