Reproduce “UniAudio-Token: Empowering Semantic Speech Tokenizers with General Audio Perception” on OpenResearch

About this paper

Yuhan Song, Linhao Zhang, Aiwei Liu, Chuhan Wu, Sijun Zhang, Wei Jia, Yuan Liu, Houfeng Wang and 1 more

Peking University and Tencent researchers introduced UniAudio-Token, a single-codebook audio tokenizer that integrates general audio perception into semantic speech tokenizers. The framework enhances disentanglement of acoustic features, achieves superior speech reconstruction fidelity with a 3.68% WER and 4.19 pMOS, and improves universal audio understanding (61.10% accuracy on MMAU) and controllable text-to-speech synthesis.

Key insights

• Structured supervision, like SAP, effectively disentangles linguistic and acoustic features, allowing a single tokenizer to support both high-fidelity speech generation and comprehensive general audio understanding.
• The SAE mechanism acts as a content-aware adaptive controller, dynamically regulating the flow of shallow acoustic information to balance linguistic abstraction with acoustic detail based on input modality and noise.
• Retaining fine-grained acoustic cues through adaptive fusion not only empowers general audio perception but also improves speech reconstruction quality by enabling clearer phonetic realization.

Summary by alphaXiv.