Reproduce “In-Context Learning Enables Robot Action Prediction in LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yida Yin, Zekai Wang, Yuvan Sharma, Dantong Niu, Trevor Darrell, Roei Herzig
The RoboPrompt framework enables off-the-shelf text-only Large Language Models to directly predict robot actions through in-context learning, bypassing the need for dedicated training or finetuning on robotic data. It achieves an average success rate of 51.8% in simulation and over 80% in real-world tasks, outperforming zero-shot and in-context learning baselines.
Key insights
- Off-the-shelf, text-only LLMs can effectively predict continuous 6-DoF robot actions directly via In-Context Learning without any dedicated training or finetuning on robotic data.
- Strategic keyframe extraction and the meticulous discretization and textualization of continuous robot observations and actions are critical for successfully leveraging LLM's ICL capabilities.
- LLMs demonstrate emergent compositional learning, capable of performing complex multi-stage tasks by inferring how to sequence individual skills from single-stage ICL demonstrations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.