Reproduce “Evaluation of Automatic Speech Recognition Using Generative Large Language Models” on OpenResearch

About this paper

Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil, Sergio Burdisso, Petr Motlicek, Shiran Liu, Mickael Rouvier, Jane Wottawa and 1 more

Researchers at Idiap Research Institute systematically evaluated generative Large Language Models (LLMs) for Automatic Speech Recognition (ASR) assessment, demonstrating their superior agreement with human perception in identifying better transcription hypotheses (up to 94%) compared to traditional and encoder-based semantic metrics. The study also showed LLMs can qualitatively classify ASR errors and identified effective pooling strategies for LLM-derived embeddings in semantic distance calculations.

Key insights

• Generative LLMs exhibit a high degree of agreement with human judgment in selecting superior ASR hypotheses, significantly outperforming traditional metrics and encoder-based semantic metrics like BERTScore/SemDist.
• For semantic similarity measurement using decoder LLM embeddings, simple mean pooling often outperforms other strategies, including the less effective 'last token' pooling, particularly for generic LLMs not explicitly trained for embeddings.
• LLMs specifically fine-tuned for embedding tasks (e.g., Qwen3-Embedding-8B) consistently yield higher quality embeddings for semantic distance calculations compared to generic generative LLMs of similar size.
• Generative LLMs can consistently and meaningfully classify ASR errors into qualitative categories, with these classifications showing clear alignment with independent quantitative semantic distance scores, enhancing interpretability.

Summary by alphaXiv.