Reproduce “UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuxuan Chen, Peize He, Haoyuan Yu, Junzi Zhang
UniWhisper develops a single, robust audio encoder that effectively processes speech, environmental sounds, and music through an efficient continual multi-task training framework. The model achieved normalized weighted averages of 0.81 with MLP probes and 0.61 with kNN evaluation across a 20-task benchmark, surpassing the Whisper Large-v3 baseline and other specialized models while preserving strong speech recognition capabilities.
Key insights
- Unifying supervision through a prompt-guided continual multi-task training framework can effectively expand an audio encoder's capabilities across speech, environmental sound, and music.
- Replacing the original Whisper decoder with a frozen, compact pretrained language model (Qwen3-0.6B) accelerates convergence and better preserves the structure of audio representations due to its strong language priors.
- A single encoder design, when appropriately trained, can overcome the limitations of domain-specific encoders and complex multi-encoder systems without architectural redundancy or token overhead.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.