Reproduce “Universal Visual Decomposer: Long-Horizon Manipulation Made Easy” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zichen Zhang, Yunshuang Li, Osbert Bastani, Abhishek Gupta, Dinesh Jayaraman, Yecheng Jason Ma, Luca Weihs

The Universal Visual Decomposer (UVD) method automatically segments long-horizon robotic manipulation tasks into meaningful subgoals by detecting "phase shifts" in embedding distances from pre-trained visual representations. This approach significantly enhances compositional generalization in imitation learning and facilitates efficient reinforcement learning across both simulated and real-world complex manipulation scenarios.

Key insights

  • Pre-trained visual representations capture temporal task progress such that their embedding distances to a goal are roughly monotonic for atomic skills but exhibit identifiable 'phase shifts' at transitions between distinct subtasks in long-horizon videos.
  • These 'phase shifts' can be automatically detected as unsupervised signals for semantically meaningful subgoal discovery without requiring task-specific knowledge or additional training.
  • Repurposing existing, frozen visual encoders provides a computationally lightweight and generalizable method for task decomposition, enhancing both imitation and reinforcement learning paradigms.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.