Reproduce “Dense Motion Captioning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shiyao Xu, Benedetta Liberatori, Gül Varol, Paolo Rota
Researchers from the University of Trento and LIGM introduce Dense Motion Captioning (DMC), a new task for 3D human motion understanding that requires localizing and describing multiple atomic actions within complex, untrimmed motion sequences. They present CompMo, a large-scale dataset with precise temporal and textual annotations, and DEMO, a novel LLM-based model that significantly outperforms baselines in dense captioning quality and temporal localization.
Key insights
- Preliminary experiments confirmed that existing single motion captioning models perform poorly on complex motions, validating the need for a dense captioning task.
- A continuous motion representation, as employed by DEMO's motion adapter, significantly outperforms discrete tokenization methods (like VQ-VAE) for capturing the nuances of long, fine-grained motions.
- The two-stage training strategy (initial motion-language alignment followed by dense captioning instruction tuning) is critical for achieving robust performance in both motion understanding and text generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.