Reproduce “Fish2Mesh Transformer: 3D Human Mesh Recovery from Egocentric Vision” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
David C. Jeong, Aditya Puranik, James Vong, Vrushabh Abhijit Deogirikar, Ryan Fell, Julianna Dietrich, Maria Kyrarini, Christopher Kitts
Fish2Mesh Transformer accurately recovers 3D human meshes from egocentric fisheye camera images by incorporating a novel Egocentric Position Embedding to account for lens distortions. The system consistently surpasses state-of-the-art methods in precision, achieving a 11.2% reduction in PA-MPJPE on the ECHP dataset compared to EgoHMR, and demonstrates improved robustness through prompt-based data augmentation.
Key insights
- Explicitly embedding 3D spatial context derived from equirectangular projection into a learnable position embedding effectively mitigates fisheye lens distortions within a transformer architecture.
- A multi-component loss function, combining SMPL, orientation, and joint losses, ensures robust geometric consistency for accurate egocentric 3D human mesh recovery.
- Augmenting datasets with naturalistic, prompt-driven actions, including those causing self-occlusions, significantly enhances a model's ability to generalize to real-world egocentric scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.