Reproduce “Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Nithin Rao Koluguri, Sasha Meister, Nikolay Karpov, Piotr Zelasko, Desh Raj, Jagadeesh Balam, Boris Ginsburg

Preference-ASR benchmarks whether speech recognizers follow requested output styles by pairing audio with explicit instructions and preference-conditioned references. Its selective scoring preserves distinctions that standard normalization can erase, covering normalization, entities, disfluencies, and case. Across four models, Qwen3-Omni-30B’s disfluency WER falls from 10.83% to 9.90% with instruction, while entity WER rises under prompting on this benchmark.

Key insights

  • SpeechLLMs can demonstrate effective compliance for specific formatting instructions (e.g., normalization, disfluencies) when properly tuned, but instruction prompts can also induce entity hallucination if they conflict with acoustic evidence.
  • An LLM backbone alone is not sufficient for instruction-following ASR; models require specific preference-aligned training to effectively leverage natural-language instructions.
  • Standard WER evaluation, due to its blanket normalization, obscures true model capabilities and limitations in adhering to user-specified output formats, which the preference-aware WER effectively reveals.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.