Reproduce “LASER: An LLM-based ASR Scoring and Evaluation Rubric” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Amruta Parulekar, Preethi Jyothi
Researchers at the Indian Institute of Technology Bombay developed LASER, an LLM-based ASR evaluation rubric, to provide a more linguistically sensitive assessment for languages with complex morphological structures, such as Hindi. This metric achieved a 94% Pearson's correlation with human judgments for Hindi, outperforming traditional metrics, and demonstrated cross-lingual transferability to other Indic languages and English.
Key insights
- LLMs, when provided with a meticulously crafted prompt and rich examples, can perform complex, fine-grained linguistic evaluations with high accuracy and demonstrate reasoning capabilities beyond explicitly provided rules.
- A human-aligned, rule-based evaluation metric tailored for morphological richness can significantly outperform traditional edit-distance and semantic similarity metrics in correlating with human perception of ASR quality.
- The evaluation framework, initially developed for one language, can effectively transfer its underlying logic to evaluate ASR outputs in other languages, including those from different language families, demonstrating LLM generalization capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.