Reproduce “CoMemo: LVLMs Need Image Context with Image Memory” on OpenResearch

About this paper

Shi Liu, Weijie Su, Xizhou Zhu, Wenhai Wang, Jifeng Dai

Chinese Academy of Sciences and Shanghai AI Lab researchers develop CoMemo, a dual-path architecture that addresses the "lost in the middle" phenomenon and positional encoding limitations in Large Vision-Language Models by combining a Context path (autoregressive processing) with an Image Memory path (cross-attention retrieval) alongside RoPE-DHR positional encoding that preserves 2D spatial relationships in high-resolution images, achieving 17.2% relative improvement on caption tasks and 7.0% on long-generation tasks compared to single-paradigm baselines while demonstrating superior performance across seven benchmark categories including multi-image reasoning where CoMemo maintains higher accuracy for information placed in middle sequences compared to conventional LVLM-S models, with the framework employing a three-stage training strategy that prevents optimization bias between pathways through gate freezing and validates consistent improvements at both 2B and 8B parameter scales across diverse fine-tuning datasets.

Key insights

• Both LVLM-X and LVLM-S paradigms have complementary strengths that can be effectively combined through dual-stream processing
• Positional encoding compression and 2D spatial preservation are crucial for handling dynamic high-resolution images in vision-language models
• Training stage management prevents over-reliance on single pathways and ensures balanced multimodal representation learning

Summary by alphaXiv.