Reproduce “Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yanhao Jia, Ji Xie, S Jivaganesh, Hao Li, Xu Wu, Mengmi Zhang
This research reveals that current AI models for sound source localization exhibit strong visual biases and struggle with sensory conflicts, unlike human perception. The study introduces AudioCOCO, a high-fidelity dataset, and EchoPin, a neuroscience-inspired model incorporating HRTFs and cochleagrams, which significantly improves sound localization accuracy and demonstrates human-like horizontal-vertical spatial asymmetry.
Key insights
- AI models exhibit a pronounced visual bias, struggling to suppress misleading visual cues and performing poorly in auditory-only localization tasks, a fundamental difference from robust human auditory perception.
- Incorporating biologically plausible auditory processing mechanisms (HRTF-based filtering and cochleagrams) significantly enhances AI models' robustness and leads to emergent human-like spatial reasoning, such as horizontal-vertical localization asymmetry.
- The quality and unbiased nature of training data, particularly high-fidelity stereo audio with depth information, are critical for developing multimodal AI models that genuinely integrate cross-modal information rather than relying on visual shortcuts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.