Reproduce “Auditory Illusion Benchmark for Large Audio Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hayoon Kim, Eunice Hong, Kyogu Lee
The Auditory Illusion Benchmark (AIB) evaluates whether large audio-language models reproduce human perceptual biases rather than merely identifying the physical contents of a sound. Across 14,829 illusion and control stimuli, current models remained mostly literal on physics-based illusions but showed greater human-like behavior for language- and music-dependent effects; Audio Flamingo 3 achieved the highest average illusion susceptibility index at 0.455, while prompt changes shifted scores by up to 0.6.
Key insights
- Most evaluated models behave more like literal acoustic detectors than human listeners on primarily physics-based illusions; only MuLLaMa and Audio Flamingo 3 showed positive average alignment in this category, with ISI scores of 0.319 and 0.271.
- Human-like responses were more common for physics-plus-knowledge illusions involving language, music, and learned expectations, suggesting that some alignment comes from high-level priors rather than shared auditory mechanisms.
- Illusion susceptibility is not an unconditional quality measure: control accuracy is essential, and prompt wording can substantially change the apparent perceptual profile of a model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.