Reproduce “EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz, Oluwanifemi Bamgbose, Fanny Riols, Hoang H. Nguyen, Raghav Mehndiratta, Lindsay Devon Brin and 5 more
EVA-Bench evaluates voice agents in multi-turn audio conversations across three synthetic English-language enterprise domains. It validates simulated caller behavior before scoring task accuracy and conversational experience over five trials per scenario. In clean conditions, none of 12 systems exceeded 0.5 on both EVA-A and EVA-X pass@1; GPT-Realtime-1.5 scored 0.47 and 0.57, respectively. Median pass@k–pass^k gaps were 0.44 and 0.24, respectively, limiting conclusions to tested settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.