Reproduce “VO-DP: Semantic-Geometric Adaptive Diffusion Policy for Vision-Only Robotic Manipulation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zehao Ni, Yonghao He, Lingfeng Qian, Jilei Mao, Fa Fu, Wei Sui, Hu Su, Junran Peng and 2 more
VO-DP-1 combines semantic features from DINOv2 with geometry-aware features from VGGT, then fuses them with robot state to predict actions from RGB observations. Across RoboTwin’s 14 simulated tasks, it averages 64.6% success, versus DP3’s 64.0%. Across four real-world tasks, it averages 87.9%, versus DP3’s 67.5%; these results apply only to the tested settings.
Key insights
- Vision-only visuomotor policies, when augmented with sophisticated feature extraction from pretrained visual foundation models, can achieve performance equivalent to or better than explicit 3D-sensing methods.
- Implicit 3D understanding derived from 2D RGB images through models like VGGT can effectively overcome traditional limitations of vision-only methods in handling scene geometry and variations.
- Adaptive fusion of semantic and geometric features is crucial for building robust and generalizable representations for diverse manipulation tasks, enhancing the model's ability to prioritize relevant visual cues.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.