Reproduce “Investigating and Enhancing Vision-Audio Capability in Omnimodal Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rui Hu, Delai Qiu, Shuyu Wei, Jiaming Zhang, Yining Wang, Shengping Liu, Jitao Sang
Researchers investigated the performance disparity between vision-text and vision-audio capabilities in omnimodal large language models (OLLMs), identifying weaker vision-audio modality alignment and attention discrepancies as key causes. A Self-Knowledge Distillation framework was developed, enhancing OLLM vision-audio performance by up to 12.74 average points on benchmarks and aligning internal attention mechanisms more effectively.
Key insights
- OLLMs display lower attention to vision tokens and higher self-attention to audio query tokens in middle and later layers compared to text queries, contributing to integration failure.
- A newly introduced MMAlign benchmark confirmed that vision-audio alignment within OLLMs is significantly weaker than vision-text alignment across various models.
- The inherent vision-language capability of the base OLLM directly correlates with the effectiveness of the Self-KD framework in improving vision-audio performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.