Reproduce “GEM: Generative Supervision Helps Embodied Intelligence” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ruowen Zhao, Bangguo Li, Zuyan Liu, Yinan Liang, Junliang Ye, Fangfu Liu, Diankun Wu, Zhengyi Wang and 4 more

GEM trains a vision-language model to generate scene depth maps alongside language tasks, aiming to encode geometry in visual representations. Its staged training aligns depth generation with language learning, and GEM-4M supplies embodied grounding, spatial reasoning, and planning data. On task-specific real-world UR5 evaluation, GEM-VLA reports 43.0% average success versus 40.7% without depth supervision, suggesting a benefit in those tested tasks.

Key insights

  • Integrating depth map generation as a generative supervision task during VLM pre-training compels the model's visual backbone to learn physically-grounded representations that encode fine-grained structural details and geometric relationships.
  • This generative supervision simultaneously enhances both low-level physical grounding and high-level semantic reasoning within a unified VLM backbone, providing reciprocal benefits.
  • A progressive training strategy is crucial for stable convergence and effective fusion of semantic and structural features, mitigating potential modality interference between the VLM backbone and the generative head.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.