Reproduce “Audio-Visual LLM for Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fangxun Shu, Lei Zhang, Hao Jiang, Cihang Xie
The Audio-Visual LLM (AV-LLM) developed by researchers from Alibaba Group, Zhejiang University, and UCSC enhances video understanding by synergistically processing visual and auditory inputs. The model improves performance on video QA (e.g., +4.4% on MSRVTT-QA) and audio-visual QA tasks (e.g., +15.9% on AVSD) through a modality-augmented training strategy and a high-quality, GPT-4 derived instruction dataset.
Key insights
- Modality-Augmented Training (MAT) effectively enhances cross-modal alignment and multimodal video understanding by jointly training with a mixture of visual-only, audio-only, and audio-visual samples.
- Integrating both visual and auditory modalities consistently improves performance in video understanding tasks compared to using visual-only data, confirming the necessity of joint audio-visual learning.
- High-quality, GPT-4 derived instruction datasets are crucial for effectively fine-tuning LLMs to perform complex, multi-modal reasoning and instruction-following in video understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.