Reproduce “Learning to Detect and Track Visible and Occluded Body Joints in a Virtual World” on OpenResearch

About this paper

Matteo Fabbri, Fabio Lanzi, Simone Calderara, Andrea Palazzi, Roberto Vezzani, Rita Cucchiara

Researchers introduced an end-to-end deep network called THOPA-net, designed for integrated multi-person pose estimation and short-term tracking, which explicitly handles occluded body parts. The model, trained on the newly created JTA synthetic dataset, effectively detects and tracks visible and occluded body joints, demonstrating generalization to real-world scenarios and outperforming baselines in joint detection and tracking metrics.

Key insights

• Explicitly differentiating and predicting heatmaps for occluded body parts significantly enhances overall pose estimation accuracy and aids in robustly clustering joints into complete skeletons, even when visual information is limited.
• The integration of Temporal Affinity Fields (TAFs) effectively provides short-term temporal consistency, improving body part association across frames and disambiguating targets in crowded or ambiguous situations.
• Models trained exclusively on large-scale, detailed synthetic data (like JTA) can demonstrate good generalization capabilities to real-world images, particularly for high-quality video, presenting a viable solution to real-world data scarcity challenges.

Summary by alphaXiv.