Reproduce “Affordance-Based Manipulation Planning with Text Goals and Sim-to-Real Generalisation via Real-to-Sim Image Conversion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Solvi Arnold, Rin Karashima, Tadashi Adachi, Takafumi Mochizuki, Kimitoshi Yamazaki
An affordance-based robotic manipulation planning system is introduced, interpreting text-based goals and predicting visual state changes to generate multi-step plans. The system achieves real-world transfer for varied objects and occlusions by integrating a real-to-sim image conversion module, demonstrating robust planning in physical environments.
Key insights
- Combining learned fine-grained visual effect prediction with multimodal (text and vision) goal assessment enables robust multi-step planning for complex manipulation tasks with high-level instructions.
- Real-to-sim image conversion, facilitated by generative augmentation using depth-conditioned diffusion models, provides an effective strategy for bridging the reality gap and generalizing to varied real-world objects without extensive real-world training data.
- Enriching state representations with explicit object mask channels and exploiting translational and rotational invariance in prediction significantly improves robustness to object occlusion and diverse object arrangements.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.