Reproduce “RealVLG-R1: A Large-Scale Real-World Visual-Language Grounding Benchmark for Robotic Perception and Manipulation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Linfei Li, Lin Zhang, Ying Shen
The RealVLG framework introduces a large-scale, real-world visual-language grounding and grasping benchmark (RealVLG-11B) and a Reinforcement Fine-tuned (RFT) model, RealVLG-R1, that unifies visual grounding and actionable grasp prediction from natural language. This model achieved up to 89% gIoU for visual grounding and significantly improved grasping performance, demonstrating robust zero-shot generalization and high success rates (up to 81% single-object, 79% cluttered) in real robot experiments.
Key insights
- Reinforcement Fine-tuning (GRPO/GSPO) on LVLMs is effective for learning physically plausible grasp poses, outperforming traditional token-level supervised fine-tuning due to grasping's multi-solution nature.
- A meticulously curated, large-scale real-world dataset like RealVLG-11B, featuring high linguistic diversity, visual-language alignment, and spatial consistency, is critical for training models capable of fine-grained, language-conditioned manipulation.
- A unified model trained with RFT on a comprehensive dataset can achieve significant zero-shot generalization across both visual grounding and actionable grasp understanding in novel scenarios and real-world environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.