Reproduce “D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Suhwan Choi, Jaeyoon Jung, Haebin Seong, Minchan Kim, Minyeong Kim, Yongjun Cho, Yoonshik Kim, Yubeen Park and 2 more

The D2E framework introduces a scalable approach for vision-action pretraining by leveraging diverse desktop interactions, particularly from gaming, and demonstrates the transferability of these learned representations to embodied AI tasks. A 1B-parameter model trained with this framework achieves comparable or superior performance to significantly larger models on robotics benchmarks and real-world robot tasks.

Key insights

  • Desktop environments, especially gaming, provide a scalable and cost-effective source of rich sensorimotor data that can effectively pretrain embodied AI models, bypassing the high costs of physical robot data collection.
  • A generalist inverse dynamics model can effectively pseudo-label internet-scale unlabeled gameplay videos, demonstrating zero-shot generalization across diverse game domains.
  • Pretraining on desktop vision-action trajectories provides better-aligned and more efficient representations for embodied control, leading to improved performance and data efficiency in robot manipulation and navigation tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.