Reproduce “RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Soroush Nasiriany, Sean Kirmani, Tianli Ding, Laura Smith, Yuke Zhu, Danny Driess, Dorsa Sadigh, Ted Xiao
RT-Affordance introduces end-effector pose affordances as an intermediate representation to guide robot policies, enhancing manipulation precision and data efficiency for learning new tasks on novel objects and scenarios. The method achieved 68% success on novel object grasping and 70% on complex manipulation tasks, utilizing a hierarchical model trained with robot, web, and efficiently collected annotated image data.
Key insights
- Affordances, defined as precise end-effector poses, offer an optimal balance between the abstractness of language and the over-specificity of goal images, providing actionable guidance for fine-grained manipulation.
- Visually projecting affordance poses directly onto the observation image effectively integrates this spatial and orientational guidance into vision-language models, enhancing their control capabilities.
- Combining large web datasets with a small set of cheaply-collected, manually annotated in-domain images for affordance prediction drastically improves data efficiency and generalization for learning novel manipulation tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.