Reproduce “Transformers for One-Shot Visual Imitation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sudeep Dasari, Abhinav Gupta

Researchers at Carnegie Mellon University developed a Transformer-based architecture for one-shot visual imitation, enabling robots to learn manipulation tasks from a single video demonstration despite domain differences between demonstrator and robot. The method achieved an 88.8% task success rate, representing approximately a 2x improvement over baselines, and could match baseline performance using 3x less data.

Key insights

  • Transformer architectures, particularly their multi-head self-attention mechanisms, are uniquely effective for extracting context-driven, task-specific features from visual demonstrations, outperforming other network designs like LSTMs for one-shot imitation.
  • The self-supervised inverse dynamics loss is crucial for learning robust representations that encode underlying physics and dynamics, significantly improving test-time performance and generalization.
  • Utilizing the full temporal context from a video demonstration, rather than relying on a single goal image, is essential for the model to accurately infer the task and execute it effectively.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.