Reproduce “Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Siyuan Huang, Xiaoye Qu, Yafu Li, Tong Zhu, Zefeng He, Muxin Fu, Daizong Liu, Wei-Long Zheng and 1 more
Persistent Visual Memory (PVM) is introduced as a module for Large Vision-Language Models (LVLMs) to counter "Visual Signal Dilution" by providing sustained, on-demand access to raw visual evidence. PVM achieves a 4.8% accuracy improvement over baselines on multimodal benchmarks, with gains up to 27.3% on long sequences, and accelerates internal prediction convergence.
Key insights
- Visual attention mass in autoregressive LVLMs exhibits an O(t⁻¹) power-law decay with sequence length (t), leading to a "Low-Attention Equilibrium" where visual signals become statistically insignificant.
- A parallel memory pathway with independent attention normalization, isolated from the main reasoning backbone, can effectively mitigate visual signal suppression without disrupting linguistic coherence.
- PVM accelerates the model's internal prediction convergence by actively offloading visual retrieval, allowing the main reasoning path to transition faster from perception to reasoning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.