Reproduce “Astra: Efficient Transformer Architecture and Contrastive Dynamics Learning for Embodied Instruction Following” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yueen Ma, Dafeng Chi, Shiguang Wu, Yuecheng Liu, Yuzheng Zhuang, Irwin King

Astra is an efficient Transformer architecture for embodied instruction following, introducing trajectory attention and learnable action queries to better process multimodal sequences. It also incorporates Contrastive Dynamics Learning (CDL) for enhanced environmental understanding and multimodal alignment. The model achieves superior performance across challenging benchmarks, including a 70.67% success rate on unseen ManiSkill tasks and faster convergence on VIMA-Bench, by more effectively leveraging multimodal trajectory structure and learning dynamics.

Key insights

  • Modifying Transformer attention masks to allow bidirectional attention *within* multimodal segments (e.g., visual observations from multiple cameras, multi-dimensional actions) significantly improves contextual understanding of embodied instruction following trajectories.
  • Dedicated, learnable action queries can more efficiently extract relevant information for predicting specific action dimensions in parallel, outperforming sequential autoregressive generation.
  • A lightweight, encoding-based contrastive learning objective can effectively enhance a model's internal representation of environmental dynamics and improve multimodal alignment without requiring complex generative modules.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.