Reproduce “Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs” on OpenResearch

About this paper

Junhao Shi, Siyin Wang, Xiaopeng Yu, Li Ji, Jingjing Gong, Xipeng Qiu

A framework called Task-Agnostic Pretraining (TAP) teaches Vision-Language-Action models fundamental physical competence from abundant unlabeled data using inverse dynamics, then grounds these priors with minimal expert demonstrations. This approach achieved an average success rate of 33.32% on the SIMPLER benchmark, outperforming foundational models trained on limited expert data, and demonstrated enhanced robustness in real-world scenarios.

Key insights

• Decomposing robot learning into task-agnostic physical competence and task-specific semantic alignment is a viable strategy to mitigate the expert data scarcity in VLA model training.
• The Inverse Dynamics objective effectively forces the visual encoder to learn dynamics-aware representations from unlabeled data, allowing the model to acquire fine-grained motor control and physical common sense without explicit task goals.
• Learning physical priors from diverse, task-agnostic exploration data significantly improves the model's robustness to visual distractors, background shifts, and camera perturbations in real-world environments.

Summary by alphaXiv.