Reproduce “Creative Robot Tool Use by Counterfactual Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
M. Tuluhan Akbulut, Varun Satheesh, Ahmed Jaafar, Alper Ahmetoglu, Shane Parr, Aditya Ganeshan, Shivam Vats, George Konidaris
A framework named ToolAnalogy enables robots to creatively repurpose novel objects as tools by combining Vision-Language Model insights with physics-based counterfactual simulation to discover causally relevant features. This approach achieved up to 90% task success across diverse manipulation scenarios and provided human-interpretable justifications for tool selection.
Key insights
- Creative tool use is primarily governed by a small set of task-dependent, causally relevant physical features, rather than superficial visual similarities or local geometry.
- Integrating the broad commonsense reasoning of VLMs with precise physics-based simulation is crucial for robust, physically grounded tool selection in robots.
- Aligning skill transfer with identified causal feature dimensions provides a more functionally meaningful and robust way to adapt manipulation skills to novel objects with varying geometries.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.