Reproduce “Quantifying Cross-Modality Memorization in Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuxin Wen, Yangsibo Huang, Tom Goldstein, Ravi Kumar, Badih Ghazi, Chiyuan Zhang
This paper presents the first systematic investigation into cross-modality memorization in Vision-Language Models, quantifying how factual knowledge transfers between visual and textual modalities. It reveals a consistent performance gap and asymmetric transfer patterns between modalities, alongside the concerning finding that models can unintentionally memorize irrelevant visual personally identifiable information.
Key insights
- Cross-modality knowledge transfer in VLMs is asymmetric, with image-to-text recall consistently outperforming text-to-image recall.
- Model scaling improves overall memorization but does not fundamentally bridge the performance gap or transfer rates between modalities.
- Effective cross-modal learning and mitigation of the modality gap require explicit, in-distribution (semantically relevant) cross-modal associations during training, rather than general multimodal exposure.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.