Reproduce “VPHO: Joint Visual-Physical Cue Learning and Aggregation for Hand-Object Pose Estimation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jun Zhou, Chi Xu, Kaifeng Tang, Yuting Ge, Tingrui Guo, Li Cheng
VPHO is a framework that jointly learns 2D visual cues and 3D physical interaction forces, then aggregates multiple pose candidates to estimate 3D hand-object poses from a single RGB image. This approach achieves state-of-the-art accuracy on DexYCB and HO3Dv2 datasets, reducing object pose errors (e.g., 26.8% MCE reduction on DexYCB) and enhancing physical plausibility with high contact percentages and reduced penetration depth.
Key insights
- Reconciling visual consistency and physical plausibility in 3D hand-object pose estimation requires jointly learning and integrating both types of cues, rather than relying on them separately.
- Employing a candidate generation and two-stage aggregation strategy (visual then physics-based) is effective for refining poses, mitigating error accumulation, and ensuring physical realism.
- Modeling 3D interaction forces with friction cones and training with a semi-supervised approach can effectively provide physical cues without demanding difficult-to-obtain ground-truth force annotations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.