Reproduce “HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video” on OpenResearch

About this paper

Zicong Fan, Maria Parelli, Maria Eleni Kadoglou, Muhammed Kocabas, Xu Chen, Michael J. Black, Otmar Hilliges

Researchers at ETH Zürich and Max Planck Institute for Intelligent Systems introduced HOLD, a method for category-agnostic 3D reconstruction of articulated hands and unknown objects from monocular video. This approach, which uses a compositional neural implicit model and leverages hand-object interaction constraints, achieves significantly improved reconstruction quality, with a hand MPJPE of 24.2 mm and object Chamfer Distance of 0.4 cm2 on the HO3D dataset, while also demonstrating robust generalization to novel objects.

Key insights

• Jointly modeling the hand and object and explicitly incorporating interaction constraints (e.g., contact points) can effectively resolve depth ambiguities inherent in monocular video and improve overall reconstruction accuracy.
• A compositional implicit model, by disentangling the representation of interacting entities, allows for robust and high-quality 3D surface reconstruction of both articulated hands and arbitrary objects from 2D observations.
• An iterative refinement strategy, starting from coarse pose estimates and progressively optimizing poses with interaction-aware losses before full shape reconstruction, is crucial for achieving accurate and consistent 3D hand-object interactions.

Summary by alphaXiv.