Reproduce “Embodied Task Planning with Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, Haibin Yan
Researchers from Beijing University of Posts and Telecommunications and Tsinghua University developed TAsk Planning Agent (TaPA), a framework that enables embodied agents to generate executable action plans by integrating Large Language Model commonsense with real-world visual perception. TaPA demonstrated an average success rate of 61.11% for complex tasks in the AI2-THOR simulator, outperforming other LLMs and LMMs, and reduced hallucination and counterfactual planning failures.
Key insights
- Integrating LLM-based commonsense reasoning with visual perception through a data generation and fine-tuning approach significantly improves plan success rates for embodied agents.
- Efficient multi-view image collection strategies, specifically 'block-wise center points', optimize visual scene representation by balancing coverage and minimizing noise from object detection, which can otherwise degrade LLM planning.
- The use of large language models like GPT-3.5 for automatic multimodal dataset synthesis provides a scalable method for generating grounded task planning data, addressing a key bottleneck in embodied AI research.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.