Reproduce “Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nils Blank, Moritz Reuss, Marcel Rühle, Ömer Erdinç Yağmurlu, Fabian Wenzel, Oier Mees, Rudolf Lioutikov
NILS introduces a modular, zero-shot framework leveraging an ensemble of frozen foundation models to automatically generate high-quality natural language annotations for large-scale, uncurated robot interaction datasets. This method substantially reduces labeling costs while producing diverse and temporally consistent labels that enable competitive performance for downstream language-conditioned robot policies.
Key insights
- Intelligently combining and orchestrating multiple off-the-shelf foundation models can effectively compensate for their individual weaknesses (e.g., domain shift, temporal reasoning) to solve complex, domain-specific robot learning problems.
- A heuristic consensus approach, which synthesizes multiple object-centric and gripper-based signals, provides robust and controllable keystate detection, superior to direct VLM captioning or uniform sampling.
- By generating structured, templated prompts for an LLM based on detected object changes, the framework enables the creation of diverse, high-quality, and semantically rich language labels with adjustable granularity for robot actions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.