Reproduce “Episodic Transformer for Vision-and-Language Navigation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alexander Pashevich, Cordelia Schmid, Chen Sun
Researchers from Inria, Google Research, and Brown University introduced the Episodic Transformer (E.T.) architecture, which processes full episode histories and leverages synthetic instructions to enhance Vision-and-Language Navigation. This approach improved task success rates on the ALFRED benchmark, establishing new state-of-the-art performance, particularly for generalization to unseen environments.
Key insights
- A purely transformer-based architecture that observes the full episode history effectively addresses long-term memory challenges in compositional VLN tasks, outperforming recurrent models.
- Synthetic instructions provide a structured, domain-specific representation that significantly improves human instruction understanding and generalization, especially to unseen environments.
- Combining language encoder pretraining with joint training using synthetic data offers complementary benefits, leading to substantial performance gains in both seen and unseen environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.