Reproduce “Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructions” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Peter A. Jansen

Large language models can infer detailed visual semantic plans for embodied agents from high-level natural language instructions without direct visual input. The GPT-2 model successfully generated perfectly matching plans for 22.2% of unseen tasks, and this accuracy increased to 53.4% when minimal visual context, specifically the agent's starting location, was provided.

Key insights

  • Pre-trained language models possess an implicit commonsense understanding of physical tasks and object interactions, allowing them to generate plausible action sequences from language alone.
  • Minimal visual context, such as the agent's starting location, significantly enhances the accuracy of language-generated plans, suggesting a powerful synergy for hybrid agent architectures.
  • A substantial portion of planning errors stemmed from incorrect location predictions and inconsistencies within the crowdsourced ALFRED dataset itself, rather than fundamental object identification issues.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.