Reproduce “History-Aware Visuomotor Policy Learning via Point Tracking” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jingjing Chen, Hongjie Fang, Chenxi Wang, Shiquan Wang, Cewu Lu
Researchers at Shanghai Jiao Tong University and Noematrix developed a history-aware visuomotor policy learning framework that leverages object-centric point tracking to create compact, structured representations of past observations. This approach enables robotic policies to handle non-Markovian tasks by effectively using long-horizon memory, achieving decision accuracies over 90% in various manipulation scenarios and yielding a 5.8x efficiency improvement with asynchronous tracking.
Key insights
- Object-centric point tracking provides a robust and compact history representation, outperforming fixed-window or keyframe-based methods, especially in long-horizon, non-Markovian tasks.
- 3D point trackers are essential for accurate history awareness in manipulation, as they effectively handle occlusions and maintain spatial relationships, significantly surpassing 2D trackers in performance.
- Asynchronous tracking combined with random dropping augmentation during training offers a practical balance between computational efficiency and high policy performance for full history-awareness.
- Selecting object-centric points for tracking is more effective than including end-effector points, which can lead to policy over-reliance and hinder task recovery in complex scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.