Reproduce “ETC-Bench: A Multilingual Benchmark for Evaluation-Side Transcript Canonicalization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiheng Qian, Junhao Du, Jing Peng, Hanqi Li, Kunyang Peng, Feng Xue, Shuai Wang, Kai Yu
The authors introduce ETC-Bench and TACS, a method for canonicalizing transcript pairs before evaluation when speech may not distinguish forms such as “Sara” and “Sarah.” TACS guides an agent to process each transcript independently, then compares the outputs with an exact-match and pair-relative semantic check. On 12 languages, TACS-Qwen scores 68.8% SENA versus 13.9% without TACS; homophonic substitutions remain difficult.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.