Reproduce “SPEECH-COCO: 600k Visually Grounded Spoken Captions Aligned to MSCOCO Data Set” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
William Havard, Laurent Besacier, Olivier Rosec
SPEECH-COCO, a dataset featuring over 600,000 visually grounded spoken captions aligned to MSCOCO images, offers 604 hours of audio data. This resource, developed with a commercial text-to-speech system, incorporates eight distinct synthetic voices and introduced speech variability, providing perfectly time-coded speech annotations for multimodal AI research.
Key insights
- High-quality synthetic speech, when generated with controlled variability across multiple voices and speech rates, can effectively simulate realistic speech characteristics for large-scale datasets.
- Direct generation via TTS provides inherently precise time-coded annotations at the word, syllable, and phoneme levels, overcoming the inaccuracies of post-hoc forced alignment methods.
- The introduced variability makes the synthetic corpus sufficiently challenging for existing speech processing tools, indicating its potential to drive advancements in areas like unsupervised term discovery.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.