Reproduce “Ponymation: Learning Articulated 3D Animal Motions from Unlabeled Online Videos” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Keqiang Sun, Dor Litvak, Yunzhi Zhang, Hongsheng Li, Jiajun Wu, Shangzhe Wu

Ponymation presents a method to learn a generative model for articulated 3D animal motions using only raw, unlabeled online videos, enabling the creation of diverse 4D animations from a single image. This approach generates 4D animations in seconds, outperforming previous text-to-4D models in motion quality and efficiency, and improves single-image 3D reconstruction by leveraging temporal consistency.

Key insights

  • Learning semantic correspondences from unstructured 2D videos using self-supervised image features (DINO-ViT) is effective for distilling complex 3D dynamics without explicit annotations.
  • A spatio-temporal transformer architecture can effectively encode and decode articulate 3D motions, capturing both spatial pose information and temporal dynamics.
  • Leveraging temporal consistency from video data during training significantly enhances the quality of both 3D reconstruction and motion generation compared to image-only approaches.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.