Reproduce “SSI-Policy: Learning Structured Scene Interfaces for Vision-Language Robotic Manipulation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kaijun Wang, Zikai Ouyang, Xuping Wu, Jinyi Hong, Wei Pan, Haibo Lu, Jia Pan, Wei Zhang and 1 more

SSI-Policy introduces a Structured Scene Interface (SSI) that unifies monocular depth features, language-grounded object layouts, and instruction-conditioned pixel trajectories from RGB inputs for robotic manipulation. The framework achieves an 80.42% average success rate on the LIBERO benchmark with only 10 demonstrations per task, representing a 15% improvement over prior methods, and demonstrates robust cross-embodiment transfer capabilities.

Key insights

  • Decoupling perception from control via a robot-agnostic Structured Scene Interface (SSI) enables high data efficiency and robust cross-embodiment transfer in robotic manipulation.
  • Jointly encoding monocular depth, language-grounded object layouts, and instruction-conditioned motion in a unified RGB-only interface provides complementary benefits for robust spatial reasoning and task execution.
  • Learning motion priors from action-free human-hand videos significantly boosts robot policy performance, suggesting SSI's effectiveness in leveraging diverse, readily available data sources.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.