Reproduce “RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chunru Lin, Haotian Yuan, Yian Wang, Xiaowen Qiu, Tsun-Hsuan Wang, Minghao Guo, Bohan Wang, Yashraj Narang and 2 more
RobotSmith, a collaborative effort by researchers from the University of Massachusetts Amherst, MIT, National University of Singapore, and NVIDIA, provides an automated framework for generative robotic tool design and use, integrating Vision-Language Models with physics-based optimization. This framework enables robots to autonomously design and utilize task-specific tools for complex manipulation, achieving an average success rate of 50.0% in diverse simulation tasks involving rigid, deformable, and fluid objects, and demonstrating successful real-world deployment.
Key insights
- Large Vision-Language Models can effectively guide the iterative design of functional tools by leveraging their implicit physical knowledge, spatial reasoning, and affordance perception.
- Joint optimization of both tool geometry and robot manipulation trajectories in a physics simulator is critical for achieving high success rates, as VLM-generated initial designs and plans often lack the precision for real-world physics.
- A parameterized tool representation, allowing for fine-grained adjustments, combined with a VLM-guided generative process, enables the creation of diverse, task-specific tools that extend beyond rigid templates.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.