Reproduce “ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation” on OpenResearch

About this paper

Arjun Bhardwaj, Maximum Wilder-Smith, Mayank Mittal, Vaishakh Patil, Marco Hutter

Researchers from ETH Zurich and NVIDIA developed ViserDex, a sim-to-real framework enabling robust monocular RGB-based in-hand reorientation of complex objects by integrating 3D Gaussian Splatting with novel pre-rasterization augmentations for visual data generation. The system demonstrated sustained reorientation across diverse objects and under challenging adversarial lighting conditions, achieving up to 37.6 consecutive reorientations and reducing computational training requirements to a single consumer-grade GPU.

Key insights

• Robust dexterous manipulation with monocular RGB is achievable by addressing the 'perception-control gap' through a computationally efficient and visually realistic sim-to-real pipeline.
• Domain randomization can be effectively performed directly within the 3D Gaussian Splatting representation space, overcoming its inherent limitations for dynamic scene rendering and providing diverse training data.
• A performance-based curriculum for reinforcement learning and a belief encoder-decoder for student distillation are crucial for efficient policy training and robustness to real-world sensor noise and partial observations.

Summary by alphaXiv.