Reproduce “What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Oier Mees, Lukas Hermann, Wolfram Burgard
A systematic study identifies crucial architectural and algorithmic components for language-conditioned robotic imitation learning from unstructured data, leading to the Hierarchical Universal Language Conditioned Policies (HULC) framework. HULC achieves state-of-the-art performance on the CALVIN benchmark, completing 28.3% of 5-instruction chain tasks and an average sequence length of 3.06 on the multi-environment split.
Key insights
- Hierarchical control, which includes a local policy with relative actions and gripper camera observations, significantly improves generalization and performance in complex tasks.
- Discrete latent plans, generated by a multimodal transformer encoder, provide a more effective inductive bias for handling non-smooth task aspects and aid generalization.
- A self-supervised contrastive loss for visual-language alignment is crucial for robustly grounding language instructions to robot perception and actions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.