Reproduce “Expressive Speech Retrieval using Natural Language Descriptions of Speaking Style” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wonjune Kang, Deb Roy
Researchers at MIT introduce a framework for expressive speech retrieval, enabling users to find speech segments based on natural language descriptions of speaking style, extending beyond content or limited emotion. The system achieves performance on par with or exceeding classification-based baselines for 22 distinct styles using open-ended queries.
Key insights
- Speech encoder models pre-trained specifically for paralinguistic or emotional features (e.g., emotion2vec) are crucial for achieving high performance in expressive speech retrieval tasks over general-purpose speech models.
- The combination of an auxiliary style classification loss and an adversarial modality discriminator is indispensable for learning discriminative, modality-invariant representations that enable effective cross-modal alignment.
- Extensive prompt augmentation using large language models is vital for the system to generalize effectively to open-ended natural language queries, ensuring robust performance with diverse user inputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.