Reproduce “asr_eval: Algorithms and tools for multi-reference and streaming speech recognition evaluation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Oleg Sedukhin, Andrey Kostin

An advanced framework for Automatic Speech Recognition (ASR) evaluation is introduced, featuring the Multi-reference Wildcard and Enhanced (MWER) alignment algorithm, the `asr_eval` open-source library, and new multi-reference datasets. This system provides a more robust and nuanced assessment of ASR models, revealing how annotation styles can generate an "illusion of metric improvement" during fine-tuning.

Key insights

  • Annotation style significantly alters ASR model fine-tuning dynamics, demonstrating an 'illusion of metric improvement' where models adapt to dataset-specific labeling biases rather than achieving true, generalizable performance gains.
  • Carefully constructed multi-reference ground truths, incorporating wildcards and alternative phrasings, are essential for accurate ASR evaluation, as 50% of short, noisy speech samples and 1-2% of words in clear speech can benefit from multi-variance.
  • The improved word-to-word alignment provided by MWER is critical not only for visual clarity but also for establishing a more stable and accurate foundation for detailed error analysis and streaming latency calculations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.