Reproduce “AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinyi Wang, Xun Yang, Yanlong Xu, Yuchen Wu, Zhen Li, Na Zhao
AffordBot introduces a unified framework for fine-grained 3D embodied reasoning, enabling agents to predict specific 3D interaction masks, motion types, and directions from natural language instructions. The system achieves state-of-the-art performance on the SceneFun3D dataset, with an AP of 15.5 and a +TD (motion type and direction) of 10.8, significantly outperforming prior baselines by integrating 3D point cloud perception with Multimodal Large Language Models.
Key insights
- Unified, instruction-conditioned reasoning for 3D affordance grounding and motion estimation is critical for truly intelligent embodied agents.
- Providing comprehensive visual context through dynamically generated surround-view images and employing active view selection significantly enhances MLLM performance in 3D tasks.
- The quality of initial 3D instance segmentation is the primary bottleneck for fine-grained embodied reasoning, indicating a key area for future research and improvement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.