Reproduce “DUET-DINO: Simultaneous Cross-View World Modeling for Latent Planning in Robot Manipulation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nisarga Nilavadi, Ralf Römer, Moritz Reuss, Michael Krawez, Tobias Jülg, Angela P. Schoellig, Rudolf Lioutikov, Wolfram Burgard
DUET-DINO plans fine-grained robot reaches by predicting both fixed workspace-camera and wrist-camera views. Frozen DINOv3 features connect through cross-attention, while separate predictors forecast each view after candidate actions; CEM refines sampled action sequences against both goal views. Across four angled-reach tasks, it achieves 72.5% success versus 55.0% for an independent dual-view baseline under simulator conditions, within 100 steps.
Key insights
- Integrating complementary visual information through cross-view conditioning in the latent space significantly enhances the model's ability to predict and plan for full 7-DoF actions.
- DINOv3 visual encoders and their action-conditioned predictors are more effective at capturing fine-grained action-induced dynamics compared to V-JEPA 2, leading to superior planning performance.
- The combination of global and gripper-centric views provides robustness against visual distribution shifts and camera perturbations, which is crucial for real-world deployment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.