Reproduce “BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kunkun Pang, Dafei Qin, Yingruo Fan, Julian Habekost, Takaaki Shiratori, Junichi Yamagishi, Taku Komura
BodyFormer synthesizes vivid and diverse 3D body gestures from speech input by employing a semantics-guided variational Transformer, which significantly outperforms state-of-the-art baselines in human-likeness and appropriateness in user studies, achieving 70.5% and 69.0% respectively on the Trinity dataset.
Key insights
- Modeling gesture stochasticity with variational inference enables the generation of diverse and non-repetitive motions.
- Mode Positional Embedding (MPE) improves realism by adapting gesture speeds and styles to different speaking modes.
- Intra-modal pre-training effectively addresses data scarcity, making Transformer architectures viable for high-quality, limited MoCap datasets.
- Fusing low-level acoustic features with high-level semantic text information is essential for generating contextually rich and appropriately synchronized gestures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.