Reproduce “Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Following” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuki Inoue, Hiroki Ohashi
Prompter is a modular embodied instruction-following system that combines explicit robot geometry and interaction constraints with GPT-Neo-based object–landmark reasoning for semantic search. On ALFRED test-unseen tasks, it reaches 41.53% success with high-level instructions and 45.32% with step-by-step instructions, while matching a trained CNN search model without requiring dedicated object-location annotations.
Key insights
- Explicit embodiment modeling is highly influential: using the correct 20 cm obstacle inflation reduces collisions to 0.24%, compared with 11.00% for FILM, and substantially improves unseen-task success.
- LLM prompting provides object–landmark estimates that are comparable to a supervised CNN on ALFRED while reducing the need for dedicated spatial annotations and allowing vocabulary expansion without retraining.
- Perception and language grounding remain the main bottlenecks: ground-truth depth and segmentation raise validation-unseen success from 50.18% to 69.79%, while language-processing and target-search failures are the most common errors.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.