Reproduce “Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Clutter” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Georgios Tziafas, Yucheng Xu, Arushi Goel, Mohammadreza Kasaei, Zhibin Li, Hamidreza Kasaei
This research introduces OCID-VLG, a new dataset for language-guided robot grasping in cluttered indoor environments, and proposes CROG (CLIP-based Referring Object Grasping), an end-to-end model that integrates visual grounding and grasp synthesis. CROG achieved 81.1% IoU for segmentation and 77.2% J@1 for grasping on OCID-VLG, outperforming multi-stage baselines by significant margins.
Key insights
- Jointly fine-tuning visual grounding and grasp synthesis, leveraging pre-trained CLIP, leads to substantially better performance than traditional multi-stage pipelines by avoiding error propagation.
- A multi-task learning objective, combining a contrastive loss for segmentation and an L1-loss for grasping, facilitates beneficial knowledge transfer between the two tasks.
- The cross-attention vision-language decoder in CROG effectively handles complex linguistic expressions, including spatial concepts, by densely aligning pixel and text tokens.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.