Reproduce “CMDAR: A Chinese Multi-scene Dynamic Audio Reasoning Benchmark with Diverse Challenges” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hui Li, Changhao Jiang, Hongyu Wang, Ming Zhang, Jiajun Sun, Zhixiong Yang, Yifei Cao, Shihan Dou and 6 more

CMDAR is a new benchmark for evaluating complex audio reasoning in Large Audio Language Models (LALMs) within dynamic, multi-scene Chinese-language environments. It demonstrates that state-of-the-art models struggle significantly with advanced reasoning tasks compared to human performance, particularly in multi-audio and open-ended scenarios.

Key insights

  • CMDAR is a highly challenging benchmark, revealing substantial limitations in the complex dynamic audio reasoning abilities of current LALMs, with even top models performing considerably below human levels.
  • A significant performance gap exists between closed-source (e.g., GPT-4o Audio) and open-source models, especially on more complex open-ended and multi-audio reasoning tasks, suggesting disparities in advanced audio perception and reasoning.
  • Reasoning errors, such as broken multi-step reasoning chains, incorrect causal judgments, and insufficient contextual information integration, are identified as the primary source of model failures on the benchmark.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.