Reproduce “PhysHanDI: Physics-Based Reconstruction of Hand-Deformable Object Interactions” on OpenResearch

About this paper

Jihyun Lee, Changmin Lee, Donghwan Kim, Tae-Kyun Kim

Researchers at KAIST developed PhysHanDI, a framework that reconstructs dense 3D interactions between hands and non-rigid objects from sparse-view RGB-D videos. It achieved superior accuracy and future prediction over baselines like PhysTwin, particularly on dense contact interactions, by leveraging physics-based simulation and demonstrating reciprocal refinement between hand and object models.

Key insights

• Using dense 3D hand geometry (from MANO) to drive physics-based object simulation results in more precise interaction force modeling, leading to more accurate object deformation and an optimized spring-mass model topology.
• The physics-based deformable object model can reciprocally refine the initial hand reconstruction through an inverse physics approach, improving hand accuracy in challenging sparse input scenarios.
• A hierarchical optimization scheme for object reconstruction, adapted from PhysTwin but with dense hand inputs, successfully models large and complex non-rigid deformations with physical plausibility.

Summary by alphaXiv.