Reproduce “SkelFormer: Markerless 3D Pose and Shape Estimation using Skeletal Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Vandad Davoodnia, Saeed Ghorbani, Alexandre Messier, Ali Etemad

SkelFormer, developed by Queen's University and Ubisoft LaForge, accurately estimates 3D human pose and shape from multi-view RGB camera inputs using a novel skeletal transformer architecture. This system achieves state-of-the-art results on the Human3.6M dataset with an MPJPE of 31.7mm, while demonstrating robustness to noise and occlusions.

Key insights

  • The Skeletal Transformer incorporates a Joint Encoder, Pose Decoder, and Shape Decoder, specifically tailored for human skeletal structure.
  • A key innovation is the skeleton-aware attention mask within the Pose Decoder, which enforces anatomical constraints by limiting attention between joints based on their kinematic tree distance.
  • SVD-based symmetric orthogonalization in the Pose Decoder ensures valid rotation matrices for joint rotations, contributing to estimation accuracy.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.