Reproduce “AudioProcessBench: Benchmark for Identifying Process Errors in Audio-Grounded Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiangyu Zhao, Junyu Yan, Yaling Shen, Zimu Wang, Yiwen Jiang, Stephanie Fong, Qingyang Xu, Jiahe Liu and 2 more
AUDIOPROCESSBENCH introduces the first benchmark for process-level error identification in audio-grounded reasoning, providing fine-grained, audio-specific error annotations for 23,497 steps across 3,872 reasoning traces. Evaluations reveal a performance gap between open-source and frontier models in identifying these errors, alongside insights into self-critique biases and the relationship between generation and critic abilities.
Key insights
- Stronger critic performance is observed in newer, reasoning-oriented models, but a notable performance gap persists between open-source models and frontier models like Gemini-3-Flash.
- Model scale does not uniformly determine critic ability, as gains in step-level verification do not always proportionally translate to improved chain aggregation for answer selection.
- Many strong reasoning-oriented critics exhibit 'self-blindness,' performing worse when evaluating their own generated reasoning traces, suggesting shared failure modes between generation and verification within the same model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.