Reproduce “Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hengzhuang Li, Xinsong Zhang, Qiming Peng, Bin Luo, Han Hu, Dengyang Jiang, Han-Jia Ye, Teng Zhang and 1 more

A framework named LaVer strengthens the intrinsic visual modeling capability of Multimodal Large Language Models by integrating direct visual supervisory signals in the LLM's latent semantic space. This approach enhances visual reasoning, improves visual attention, and yields consistent performance improvements across various multimodal benchmarks, including a 19.22% gain on OCR-Bench.

Key insights

  • The "modality imbalance issue" in MLLMs stems from insufficient direct visual supervision, leading to a loss of discriminative power in visual representations over deeper layers.
  • Masked Image Modeling (MIM) at the latent semantic level of the LLM, coupled with a student-teacher framework, offers effective direct visual supervision.
  • Direct reconstruction loss alone can lead to "visual feature inconsistency," highlighting the need for asymmetric regularization like Clipped Gram-Anchoring to encourage discriminative feature learning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.