Reproduce “OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academia” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xuelong Geng, Kun Wei, Qijie Shao, Shuiyun Liu, Zhennan Lin, Zhixian Zhao, Guojian Li, Wenjie Tian and 13 more
OSUM presents an open-source Speech Understanding Model (SULM) designed for academic use, demonstrating competitive performance across eight diverse speech tasks despite being trained on a relatively limited 50,500 hours of speech data. This initiative makes advanced speech understanding technologies more accessible and transparent for researchers operating with constrained resources.
Key insights
- High-performing, multi-task SULMs can be developed with significantly fewer resources (50,500 hours of data) than typically used by industry, challenging the prevailing resource-intensive paradigm.
- The ASR+X training strategy is effective for multi-task learning, enhancing stability, reducing resource consumption, and accelerating modality alignment within the LLM.
- Strategic data processing, including LLM-based annotation for tasks like Speech Emotion Recognition and Speaking Style Recognition, is crucial for achieving robust performance with limited annotated datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.