Reproduce “TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mathis Petrovich, Michael J. Black, Gül Varol
TMR proposes a high-performance system for text-to-3D human motion retrieval, addressing limitations of prior retrieval methods and offering a robust alternative to motion synthesis. It achieves state-of-the-art performance by combining a probabilistic motion synthesis framework with contrastive learning and a novel negative filtering mechanism, while also demonstrating emergent zero-shot moment retrieval capabilities.
Key insights
- Training a model for both synthesis and retrieval jointly significantly improves retrieval performance compared to training for retrieval alone, as the generative capability ensures a more holistic latent representation.
- The InfoNCE contrastive loss proves more effective than margin-based losses for structuring the joint latent space for text-to-motion retrieval.
- Filtering out semantically similar "wrong negatives" is crucial for robust contrastive learning in motion-language datasets, preventing the model from learning superficial distinctions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.