Reproduce “Data-Centric Lessons To Improve Speech-Language Pretraining” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Vishaal Udandarao, Zhiyun Lu, Xuankai Chang, Yongqiang Wang, Violet Z. Yao, Albin Madapally Jose, Fartash Faghri, Josh Gardner and 1 more
Researchers from Apple systematically investigated data processing and curation strategies for speech-language pretraining, focusing on enhancing Spoken Question-Answering (SQA) capabilities. They found that fine-grained audio interleaving, synthetic knowledge-rich and QA datasets, and deterministic modality sampling significantly improve SQA performance, leading to their 3.8B SpeechLM, SpeLangy, achieving an average SQA accuracy of 51.8%, outperforming larger state-of-the-art models by 10.2% to 11.1%.
Key insights
- Fine-grained interleaving of speech-text chunks from raw audio content consistently yields superior SQA performance compared to coarse-grained approaches.
- Synthetic datasets, especially those structured as explicit Question-Answering data (Quest), are highly effective in boosting SQA performance and expanding topic coverage.
- Deterministic sampling, which alternates between speech and text modalities during training, enhances cross-modal learning and leads to better modality alignment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.