Reproduce “Generating Human Motion in 3D Scenes from Text Descriptions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhi Cen, Huaijin Pi, Sida Peng, Zehong Shen, Minghui Yang, Shuai Zhu, Hujun Bao, Xiaowei Zhou
A two-stage framework from Zhejiang University and Ant Group generates human motion within 3D scenes from text descriptions, leveraging large language models for object localization and object-centric diffusion models for motion synthesis. This approach achieves the lowest goal distance (0.384) and highest human perceptual scene score (3.54) on the HUMANISE dataset, indicating more accurate and realistic human-scene interactions.
Key insights
- Decomposing the complex problem into language grounding of the target object and object-centric motion generation simplifies multimodal reasoning and improves accuracy.
- Leveraging LLMs for explicit spatial reasoning and object localization bridges the gap between natural language instructions and complex 3D scene geometry.
- Object-centric scene representations, using volumetric sensors, effectively guide generative models by focusing on relevant scene geometry and reducing overall scene complexity.
- A two-stage diffusion process, separating trajectory from full motion completion, enhances the realism and gradual transitions of generated movements.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.