Reproduce “MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yi-Jen Shih, Shih-Yun Shan Kuan, Guan-Ting Lin, Kai-Wei Chang, Siddhant Arora, Shu-wen Yang, Abdelrahman Mohamed, Shinji Watanabe and 2 more
MP-Bench evaluates whether voice agents understand group conversation, including when to speak, stay silent, and respond appropriately. It tests 927 controlled audio tasks with three simulated speakers across discussion and turn-based game scenarios. Real-time agents’ game turn-taking remains near the 50% chance baseline; GPT-Realtime scores 48.44% multiparty versus 76.0% single-speaker, within benchmark limits.
Key insights
- Realtime agents can often respond when explicitly addressed, but they struggle to infer when they are entitled or required to speak under multiparty rules; game-condition turn-taking commonly remains near the 50% chance baseline.
- Speaker attribution is central to the difficulty: transcript-only inputs perform poorly, oracle speaker labels produce near-perfect game performance, and making voices acoustically similar sharply reduces comprehension accuracy.
- The results point to a multiparty state-tracking problem rather than a general inability to follow rules, since performance improves substantially in single-speaker game ablations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.