Reproduce “MemCtrl: Using MLLMs as Active Memory Controllers on Embodied Agents” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Vishnu Sashank Dorbala, Dinesh Manocha

Researchers at the University of Maryland, College College Park, developed MemCtrl, a framework that enables Multimodal Large Language Models (MLLMs) to actively manage their memory for embodied agents. By augmenting MLLMs with a trainable memory head, the framework filters observations in real-time, boosting task completion rates for smaller models by up to 20% while significantly improving memory efficiency and reducing invalid actions on long-horizon tasks.

Key insights

  • Active, write-time memory control, inspired by human cognitive processes, effectively addresses the context window limitations and inefficiency of passive memory systems for embodied MLLMs.
  • Intelligent memory filtering allows smaller, resource-efficient MLLMs to achieve performance comparable to much larger models, democratizing embodied AI for on-device deployment.
  • The modular and model-agnostic design of the memory head facilitates easy integration with various MLLM backbones without requiring costly architectural modifications or extensive fine-tuning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.