Reproduce “Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Henghui Du, Guangyao Li, Chang Zhou, Chunjie Zhang, Alan Zhao, Di Hu
The Crab model, developed by Tsinghua University, Renmin University of China, and Tencent PCG, unifies four distinct categories of audio-visual scene understanding tasks by introducing an explicit cooperation framework. It leverages a new AV-UIE instruction-tuning dataset, which includes detailed reasoning processes, and an interaction-aware LoRA architecture to mitigate inter-task interference. The model achieves state-of-the-art performance on various benchmarks, including 41.78 cIoU on Audio Referred Image Grounding and 78.94% on MUSIC-AVQA.
Key insights
- Explicit reasoning processes integrated into instruction-tuning datasets improve models' ability to handle complex audio-visual comprehension tasks.
- The interaction-aware LoRA architecture enables a large language model to dynamically specialize different LoRA heads for specific types of audio-visual interactions (e.g., temporal, spatial, pixel-level) while maintaining cooperation.
- Tasks requiring similar processing capabilities, despite data heterogeneity, exhibit clustered dependencies on specialized LoRA heads, demonstrating an emergent, effective task decomposition.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.