Reproduce “Autonomous Character-Scene Interaction Synthesis from Text Instruction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nan Jiang, Zimo He, Zi Wang, Hongjie Li, Yixin Chen, Siyuan Huang, Yixin Zhu
Researchers from Peking University and BIGAI developed a unified framework that autonomously synthesizes multi-stage human-scene interactions, including locomotion and object manipulation, directly from natural language instructions and a goal location. This was facilitated by the introduction of LINGO, a comprehensive language-annotated MoCap dataset, and novel scene and language encoders, leading to improved collision avoidance and semantic coherence in generated animations compared to prior methods.
Key insights
- The unified framework successfully integrates locomotion, hand-reaching, and human-object interaction into a single, seamless pipeline, overcoming fragmentation in previous methods.
- The Dual Voxel Scene Encoder provides detailed, anticipatory scene awareness, enabling realistic collision avoidance and natural interaction with both static and dynamic environments.
- The Frame-embedded Text Encoder is crucial for maintaining semantic coherence over long, auto-regressively generated sequences, allowing the model to understand the temporal progression within a single natural language instruction.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.