Reproduce “Train Robots in a JIF: Joint Inverse and Forward Dynamics with Human and Robot Demonstrations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gagan Khandate, Boxuan Wang, Sarah Park, Weizhe Ni, Joaquin Palacios, Kathyrn Lampo, Philippe Wu, Rosh Ho and 2 more
A framework for robot manipulation learning, called JIF, pre-trains on multi-modal human demonstrations to learn robust latent representations and then fine-tunes with a small number of robot demonstrations. This approach achieves higher success rates and better generalization with fewer robot demonstrations, notably improving performance on contact-rich tasks through the incorporation of tactile sensing.
Key insights
- Pre-training on multi-modal human demonstrations, especially incorporating tactile feedback, significantly improves data efficiency and robustness of learned robot manipulation policies.
- A dynamics-driven approach, learning joint inverse and forward dynamics in a latent space, effectively extracts manipulation-specific representations from observation-only human data.
- The use of knowledge self-distillation in a teacher-student framework stabilizes the learning process and prevents latent collapse, leading to more meaningful and decorrelated latent features.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.