Reproduce “Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs” on OpenResearch

About this paper

Linhao Zhang, Yuhan Song, Aiwei Liu, Chuhan Wu, Sijun Zhang, Wei Jia, Yuan Liu, Houfeng Wang and 1 more

Researchers from Tencent Inc. and Peking University developed the Unified Audio Schema (UAS) to improve Audio Large Language Models' (AudioLLMs) ability to perceive fine-grained acoustic details beyond mere transcription. This structured supervision framework achieved a 10.9% absolute increase in perception accuracy on the MMSU benchmark, effectively integrating linguistic content with paralinguistic and non-linguistic cues while preserving reasoning and speech generation capabilities.

Key insights

• The significant imbalance between reasoning and perception in AudioLLMs stems from the fundamental ASR-centric nature of their training, which prioritizes linguistic content over acoustic nuance.
• Structured supervision through the UAS facilitates disentangled learning, allowing models to separate linguistic information from paralinguistic and non-linguistic acoustic features, thereby overcoming the "perceptual blind spot."
• The UAS framework is universally effective across both continuous and discrete AudioLLM architectures, enhancing fine-grained perception without degrading reasoning abilities or speech generation quality.

Summary by alphaXiv.