Reproduce “Co-speech Gesture Video Generation via Motion-Based Graph Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yafei Song, Peng Zhang, Bang Zhang
Researchers at Tongyi Lab, Alibaba Group, developed a framework for generating synchronized co-speech gesture videos by first synthesizing plausible 3D motions using a multi-modal conditioned diffusion model and then retrieving corresponding video segments from a motion graph based on these generated motions. This approach achieves state-of-the-art visual quality (CD-FVD of 170.1), motion fidelity (FMD of 21.75), and synchronization (BC of 0.931), outperforming existing generative and graph-based methods.
Key insights
- Leveraging high-level semantic features from Large Language Models (LLMs) significantly improves the contextual appropriateness and overall quality of generated gestures.
- Using diffusion-generated 3D motion as the query for a motion graph effectively bridges the gap between generative diversity and real video naturalness, addressing the many-to-many mapping challenge.
- A hybrid motion similarity metric that combines both rotational and positional distances is essential for optimal retrieval of natural motion sequences from the graph.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.