Reproduce “FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera and 1 more

A new public benchmark, FLEURS, facilitates few-shot learning evaluation of universal speech representations across 102 languages by providing n-way parallel speech and text data for tasks like ASR, LangID, and cross-modal retrieval. Initial evaluations reveal significant performance disparities, particularly for low-resource and non-Latin/Cyrillic script languages, indicating areas for future model improvement.

Key insights

  • Developing comprehensive, n-way parallel benchmarks is essential for rigorously evaluating the generalization and fairness of universal speech representations across diverse languages.
  • Multimodal pre-training provides benefits for language identification, while speech-only pre-training can offer a slight edge in pure ASR performance.
  • Significant performance gaps persist for low-resource, "unseen" languages and those with unique orthographic or acoustic properties, like CJK languages in retrieval tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.