Reproduce “AnyViewDex: View-Invariant Dexterous Manipulation from RGB Observations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Soham Patil, Om Sanjay Gunjal, Sourabh Bhosale, Arhan Chavare, Ramandeep Singh Hora, Spandan Roy
AnyViewDex trains a monocular RGB policy for dexterous manipulation using simulation-only geometric supervision. Combining multi-view contrastive learning with absolute 3D coordinate prediction improves robustness to uncalibrated camera motion, reaching 76.7% success across 480 real-robot trials and substantially outperforming RGB-only and ablated baselines.
Key insights
- Viewpoint invariance alone is insufficient: the strongest policies combine cross-view feature alignment with metric geometric grounding, whereas removing either InfoNCE or coordinate supervision causes large performance drops.
- Privileged 3D supervision can be transferred into an RGB-only policy without test-time depth, calibration, or real-world fine-tuning; a visual-only linear probe predicts object position with 4.73 cm error within the training viewpoint range.
- The representation is effective but bounded by its training distribution and spatial compression: coordinate error rises to about 14.3 cm outside the camera cone, and performance is weaker for small-object localization where global pooling loses fine detail.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.