Reproduce “SITAR: A Situational Distortion Metric for Automatic Speech Recognition Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mingzhe Lu, Jinhao Liu, Yunpeng Li, Hang Su, Lichun Fan, Jian Luan
SITAR evaluates transcript differences by their effect on the described situation and intended action. It aligns shared text into paired gaps, classifies each by consequence, and aggregates severity within a bounded score. A DeBERTa-v3-large model scores pairs in one pass without runtime LLM calls. Across three datasets, scores improve with model scaling, while ordering is weaker for expression-level differences.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.