Reproduce “ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hang Li, Fengyi Shen, Dong Chen, Liudi Yang, Xudong Wang, Jinkui Shi, Zhenshan Bing, Ziyuan Liu and 1 more

ReMem-VLA is a Vision-Language-Action (VLA) architecture that integrates dual-level recurrent memory queries and Past Observation Prediction to enable continuous retention of both short-term and long-term historical context for robotic control. The model achieved a 94.5% average success rate on memory-dependent simulation tasks and an 82.5% average success rate on real-world memory-dependent manipulation tasks, substantially outperforming existing VLA baselines.

Key insights

  • The combination of dual-level recurrent memory queries (frame-level and chunk-level) provides complementary memory capabilities, effectively capturing both immediate and long-range temporal dependencies within an episode.
  • Designing the recurrent memory update path to be gradient-free is crucial for stable and effective long-term memory propagation, preventing the degradation often seen with trainable recurrent units and truncated backpropagation.
  • An auxiliary task like Past Observation Prediction significantly boosts specific memory dimensions, such as visual memory, demonstrating a targeted approach to enhance VLA performance on visually-intensive recall tasks.
  • A slot-based streaming batching strategy efficiently trains recurrent VLA policies on variable-length episodes without data padding or truncation, making the training process scalable and robust.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.