Reproduce “HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Raymond Yu, Caelan Reed Garrett, Fabio Ramos and 4 more
HAMSTER uses a vision-language model (VLM) to propose coarse two-dimensional gripper paths, including open-close cues, from images and instructions. A separate three-dimensional policy combines these paths with depth observations and robot data to execute actions. In tests, HAMSTER+3D-DA scored 0.63 on press-button tasks, versus 0.25 for fine-tuned OpenVLA; this fractional task score does not represent wholly successful trials.
Key insights
- Hierarchical decomposition, using a 2D path as an intermediate representation, allows effective utilization of abundant 'off-domain' data to fine-tune high-level VLMs for real-world robotic generalization.
- Decoupling VLM inference from high-frequency action prediction enables the use of powerful VLMs for high-level reasoning without compromising the speed and dexterity of low-level control.
- Fine-tuning a VLM on diverse cross-domain data significantly improves its ability to generate meaningful paths for novel real-world robotic tasks, outperforming zero-shot VLM prompting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.