Reproduce “Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zixuan Jiang, Yanqiao Zhu, Peng Wang, Qinyuan Chen, Xinjian Zhao, Xipeng Qiu, Wupeng Wang, Zhifu Gao and 3 more

Researchers from multiple institutions introduce Agentic ASR, an interactive framework that iteratively refines speech transcriptions using LLM-based semantic correction based on multi-turn user feedback. They also propose Sentence-level Semantic Error Rate (S2ER), a new evaluation metric, demonstrating that the system significantly reduces semantic errors by up to 88% on various benchmarks while aligning ASR closer to human communication.

Key insights

  • Iterative, agentic correction with Large Language Models (LLMs) can effectively resolve meaning-critical ASR errors, bringing ASR systems closer to human-like interactive communication.
  • Semantic-focused evaluation, such as S2ER, is crucial for assessing interactive ASR, as conventional token-level metrics often fail to capture the true functional impact of errors or the gains from semantic correction.
  • LLM-based judges can reliably assess semantic equivalence with strong human correlation, with a majority-3 voting protocol providing an effective balance between robustness and efficiency.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.