Reproduce “Text to Robotic Assembly of Multi Component Objects using 3D Generative AI and Vision Language Models” on OpenResearch

About this paper

Alexander Htet Kyaw, Richa Gupta, Dhruv Shah, Anoop Sinha, Kory Mathewson, Stefanie Pender, Sachin Chitta, Yotto Koga and 3 more

This research introduces an end-to-end framework that translates natural language descriptions into physically assembled multi-component objects by integrating 3D generative AI with Vision-Language Models (VLMs) and robotics. The system leverages VLMs for function-aware component assignment and incorporates human-in-the-loop conversational feedback, demonstrating successful robotic assembly of complex objects with high user preference for VLM-generated designs.

Key insights

• VLMs can effectively perform zero-shot, multimodal reasoning to assign functional components, leading to more intuitive and functionally appropriate designs than rule-based or purely geometric methods.
• Integrating human-in-the-loop conversational feedback into the AI-driven design process is crucial for accommodating user preferences and fostering human-AI co-creation.
• Preprocessing the AI-generated mesh to omit inaccessible or unsuitable faces ensures that VLM outputs inherently comply with fabrication constraints for robotic assembly.

Summary by alphaXiv.