Reproduce “Whisper-Aware LLM: Self-Supervised Uncertainty Learning for Robust Whispered Speech Recognition” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gaopeng Xu, Zhenyu Wang, Zheng Xue, Yinfeng Xia, Haitao Yao
Alibaba's Qwen Business Unit developed a Whisper-Aware LLM framework that integrates self-supervised uncertainty learning into an Audio-LLM to improve whispered speech recognition and mitigate hallucination. This framework achieved a 17% relative Character Error Rate reduction on Chinese whispered speech and decreased the hallucination rate from over 25% to 4.5% on non-speech audio.
Key insights
- Explicitly teaching ASR models to quantify the intrinsic uncertainty of speech signals, particularly the absence of F0, effectively addresses the root cause of whispered speech recognition difficulties.
- Self-supervised tasks can successfully train an uncertainty perception module without requiring explicit uncertainty labels, deriving confidence from physical characteristics of the acoustic signal.
- A multi-stage training protocol and a Confidence-Fused Decoding mechanism allow for stable integration of uncertainty awareness into large pre-trained LLMs, enabling adaptive control over generative output.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.