Reproduce “Learning joint reconstruction of hands and manipulated objects” on OpenResearch

About this paper

Yana Hasson, Gül Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, Cordelia Schmid

This research introduces an end-to-end learnable model for joint 3D reconstruction of hands and manipulated objects from single RGB images. It leverages a novel differentiable contact loss to enforce physical plausibility and presents the large-scale synthetic ObMan dataset, demonstrating improved reconstruction accuracy and transferability to real-world scenarios.

Key insights

• The inclusion of a balanced differentiable contact loss significantly improves the physical quality of hand-object reconstructions by reducing interpenetration and enhancing grasp stability.
• Training models with images depicting hand-object occlusions is critical for robust reconstruction performance, yielding substantial improvements over training on isolated or unoccluded data.
• Pre-training on large-scale synthetic data like ObMan enables effective transfer to real-world datasets, particularly in low-data regimes, and benefits from matching the synthetic data's distribution to the target real-world distribution.

Summary by alphaXiv.