Reproduce “Towards Open Domain Text-Driven Synthesis of Multi-Person Motions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mengyi Shan, Lu Dong, Yutao Han, Yuan Yao, Tao Liu, Ifeoma Nwogu, Guo-Jun Qi, Mitch Hill
A framework enables text-driven synthesis of multi-person motions for an arbitrary number of subjects by creating and leveraging large-scale datasets of text-annotated poses and motions from in-the-wild media, outperforming prior two-person motion generation methods.
Key insights
- Leveraging readily available 'in-the-wild' image and video data, coupled with modern pose estimation and AI-based captioning, provides a scalable alternative to expensive multi-person motion capture for dataset creation.
- Decomposing the learning into dedicated pose-focused (spatial interaction) and motion-focused (temporal coherence) layers within a transformer architecture effectively manages the complexity of multi-person dynamics.
- A two-stage training strategy, where robust pose conditioning is learned first and then preserved while motion dynamics are introduced, is crucial for synthesizing both plausible group configurations and fluid animations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.