Reproduce “Pixel Motion as Universal Representation for Robot Control” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kanchana Ranasinghe, Xiang Li, E-Ro Nguyen, Cristina Mata, Jongwoo Park, Michael S Ryoo
LangToMo presents a dual-system framework for language-conditioned robot control, leveraging dense pixel motion as an embodiment-agnostic intermediate representation. The method learns from unlabelled video-caption data, achieving 71.3% success in real-world tasks and 57.7% in simulated environments, outperforming several existing approaches.
Key insights
- Dense pixel motion (optical flow) serves as an effective, universal, and interpretable representation for robot actions, facilitating knowledge transfer across different manipulators and human-to-robot learning.
- A hierarchical dual-system architecture efficiently decouples high-level motion planning from low-level action execution, allowing sparse invocation of expensive generative models and dense, real-time control.
- Conditional diffusion models are crucial for accurately modeling the multi-modal output space when mapping abstract language instructions to concrete future motion sequences.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.