Reproduce “An Embodied Generalist Agent in 3D World” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu and 2 more
Researchers at the Beijing Institute for General Artificial Intelligence (BIGAI) introduce LEO, an embodied multi-modal generalist agent that perceives, grounds, reasons, plans, and acts in 3D environments. It achieves performance comparable to or exceeding task-specific models across 3D vision-language understanding, scene-grounded dialogue, planning, robotic manipulation, and object navigation tasks.
Key insights
- An object-centric 3D representation provides a simple yet effective method for connecting 3D scenes with large language models, leveraging their prior knowledge efficiently.
- Training with a generalist-style instruction tuning approach across a broad range of scenes and tasks significantly improves generalization compared to specialist models.
- The initial 3D vision-language alignment stage is crucial for improving downstream task performance, particularly for detailed scene understanding and captioning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.