Reproduce “SemiMultiPose: A Semi-supervised Multi-animal Pose Estimation Framework” on OpenResearch

About this paper

Ari Blau, Christoph Gebhardt, Andres Bendesky, Liam Paninski, Anqi Wu

SemiMultiPose extends DirectPose for multi-animal pose estimation by using bounding-box and keypoint-vector-field branches to create pseudo-keypoints for unlabeled images and train keypoint heatmaps. In honeybee Train5, test AP was 0.776 ± 1.45e−2, versus 0.743 for its labeled-only control and 0.734 for DirectPose. The result supports a benefit in this five-labeled-frame setting; benefits may be small when labels are abundant.

Key insights

• Unlabeled video data in animal behavioral studies contains valuable structural information that, when effectively utilized, can significantly enhance multi-animal pose estimation accuracy and training efficiency.
• Differentiable pseudo-label generation for multi-instance pose estimation can be achieved by integrating predictions from object localization and keypoint regression branches, circumventing the non-differentiability issues of soft-argmax in multi-modal contexts.
• The inclusion of consistency-promoting fusion losses not only improves the model's predictive power but also accelerates training convergence and makes the model more robust to limited or noisy labeled datasets.

Summary by alphaXiv.