Reproduce “MuLA-Bench: A Multilingual Long-Form Audio Understanding Benchmark via Multi-Tier Auditing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zeyu Yang, Xinyu Zhang, Zibo Bi, Pei Zhang, Xize Cheng, Jin Xu, Baosong Yang, Satoshi Nakamura
MuLA-Bench evaluates long-form audio-language models across 1,769 recordings, 16 languages, eight domains, and 5,038 evidence-grounded questions. By separating semantic and natural-acoustic tasks, temporal ordering from clock-time localization, and language effects from domain effects, it shows that models can retrieve distant speech more reliably than they can recognize or precisely time non-speech events.
Key insights
- Long-range spoken-information retrieval is not a sufficient proxy for general long-form audio understanding: models generally performed much better on long-range semantic questions than on acoustic or temporal tasks.
- Language performance changes substantially by domain. For example, Vietnamese had a roughly +14.2 percentage-point residual in entertainment but −13.3 points in product content, showing that pooled language rankings can be misleading.
- Recognizing an acoustic event and placing it at the correct clock time are distinct capabilities; models often identified events correctly while reporting inaccurate timestamps.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.