Reproduce “From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Dongsheng Jiang, Yuchen Liu, Songlin Liu, Jin'e Zhao, Hao Zhang, Zhen Gao, Xiaopeng Zhang, Jin Li and 1 more

This research provides an extensive investigation into the effectiveness of different visual encoders and multi-level features for Multi-modal Large Language Models (MLLMs). The paper introduces COMM, a model that fuses multi-level features from both CLIP and DINOv2, which leads to enhanced fine-grained perception, improved grounding abilities, and reduced object hallucination across various vision-language tasks.

Key insights

  • Different layers of visual encoders like CLIP exhibit distinct biases; shallow layers are effective for fine-grained tasks, while deep layers are better for global understanding.
  • Vision-only models such as DINOv2, when aligned with a non-linear Multi-Layer Perceptron (MLP), can serve as effective MLLM visual branches and even surpass CLIP in fine-grained perception tasks.
  • Merging multi-level features from visual encoders, particularly using an LLN-Layerscale strategy, significantly enhances MLLM performance across various vision-language tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.