Reproduce “TinyAlign: Boosting Lightweight Vision-Language Models by Mitigating Modal Alignment Bottlenecks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuanze Hu, Zhaoxin Fan, Xinyu Wang, Gen Li, Ye Qiu, Zhichao Yang, Wenjun Wu, Kejian Wu and 3 more

Researchers from Beihang University and collaborators developed TinyAlign, a framework that enhances lightweight Vision-Language Models by addressing a modal alignment bottleneck through an internal retrieval-augmented generation mechanism. This approach improves training efficiency, boosts performance on multimodal benchmarks, and allows comparable results with up to 60% less instruction tuning data.

Key insights

  • The performance ceiling of lightweight VLMs is constrained by an "Effective Mutual Information (EMI) bottleneck," where the limited capacity of frozen LLMs restricts the information they can practically leverage for alignment.
  • Strategically augmenting the LLM's input with "LLM-assimilable contextual hints" from an internal memory bank can effectively increase EMI, thereby reducing the irreducible error and improving multimodal alignment.
  • Careful design of the memory bank, including compact key-value generation and consistency between vision encoders used for VLM training and RAG key generation, is crucial for optimal performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.