Reproduce “MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xiyu Ren, Zhaowei Wang, Yiming Du, Zhongwei Xie, Chi Liu, Xinlin Yang, Haoyue Feng, Wenjun Pan and 6 more

MEMLENS introduces a benchmark for evaluating multimodal long-term memory in large vision-language models and memory-augmented agents, providing a length-controlled comparison across five memory abilities. The evaluation reveals that long-context LVLMs suffer degradation with increased context length, while memory agents lose visual fidelity but maintain length stability.

Key insights

  • Long-context LVLMs exhibit significant performance degradation as multimodal conversational histories grow, particularly impacting hallucination control and retrieval accuracy.
  • Memory-augmented agents demonstrate superior length stability but experience a consistent loss of fine-grained visual fidelity due to compression during storage.
  • Multi-Session Reasoning is the most challenging memory ability for both paradigms, with failures primarily stemming from upstream evidence retrieval rather than intrinsic reasoning deficits.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.