Reproduce “VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models” on OpenResearch

About this paper

Xinlei Yu, Chengming Xu, Guibin Zhang, Zhangquan Chen, Yudong Zhang, Yongbo He, Peng-Tao Jiang, Jiangning Zhang and 2 more

The VisMem framework enhances Vision-Language Models by integrating dynamic short-term and long-term latent vision memories, drawing inspiration from human cognitive memory systems. This approach yields an 11.0% average performance improvement across 12 diverse visual understanding, reasoning, and generation benchmarks, while also improving cross-domain generalization and alleviating catastrophic forgetting with minimal added inference latency.

Key insights

• Human cognitive memory theory, specifically the distinction between short-term and long-term memory, offers a robust framework for enhancing VLM visual processing.
• Dynamically injecting distinct types of latent vision memories (perceptual and semantic) can directly address the visual processing bottleneck in VLMs without altering core parameters.
• A two-stage reinforcement learning approach can effectively optimize both the formation of high-quality memory content and the strategic, self-adaptive invocation of these memories.

Summary by alphaXiv.