Reproduce “MNV-17: A High-Quality Performative Mandarin Dataset for Nonverbal Vocalization Recognition in Speech” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jialong Mai, Jinxin Ji, Xiaofen Xing, Chen Yang, Weidong Chen, Jingyuan Xing, Xiangmin Xu
The MNV-17 framework provides a 7.55-hour Mandarin speech dataset featuring 17 distinct and balanced nonverbal vocalization categories for developing expressive Automatic Speech Recognition (ASR). Benchmarking revealed that large, multi-task, autoregressive audio models achieve the best joint lexical and nonverbal vocalization recognition, with one model (Qwen2-Audio) demonstrating a 0.45% improvement in lexical Character Error Rate when nonverbal vocalizations were integrated.
Key insights
- Large, multi-task, autoregressive audio models are particularly effective for joint ASR and nonverbal vocalization recognition, leveraging their flexibility to integrate both lexical and non-lexical events.
- Integrating nonverbal vocalization recognition can enhance core lexical ASR performance in certain models (Qwen2-Audio's CER improved by 0.45%), suggesting NVs provide valuable contextual cues.
- Multi-task pretraining on diverse audio data is crucial for models to develop generalized acoustic representations necessary for robust nonverbal vocalization recognition, as evidenced by an ASR-exclusive model's lower performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.