Reproduce “TSTMotion: Training-free Scene-aware Text-to-motion Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ziyan Guo, Haoxuan Qu, Hossein Rahmani, Dewen Soh, Ping Hu, Qiuhong Ke, Jun Liu

The TSTMotion framework enables realistic human motion generation that interacts appropriately with 3D scenes from text descriptions, operating entirely without specific scene-aware training. It achieves this by adapting pre-trained blank-background motion diffusion models, showing improved motion quality, scene interaction, and collision avoidance on the HUMANISE and AffordMotion benchmarks, alongside greater diversity and generalizability to various scene types.

Key insights

  • Pre-trained blank-background motion diffusion models can be effectively made scene-aware without additional training by applying gradient-based modifications during inference.
  • Large Language Models (LLMs) are capable of high-level spatial reasoning from simplified 3D scene representations and generating actionable motion guidance.
  • Integrating a Scene Compiler to process raw 3D scene data, an LLM-based Motion Planner, and an Aligned Motion Diffusion Model provides a comprehensive, training-free approach to scene-aware motion generation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.