Reproduce “MedMosaic: A Challenging Large Scale Benchmark of Diverse Medical Audio” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Harshit Rajgarhia, Shuubham Ojha, Asif Shaik, Akhil Pothanapalli, Rachuri Lokesh, Abhishek Mukherji, Prasanna Desikan
MedMosaic, a large-scale benchmark, introduces 46,701 question-answer pairs from diverse medical audio to evaluate advanced reasoning capabilities of language and audio models in clinical settings. The benchmark reveals that even leading models like Gemini-2.5-Pro achieve a weighted average accuracy of 68.1%, indicating considerable room for improvement in medical audio reasoning.
Key insights
- Current state-of-the-art models struggle significantly with complex medical audio reasoning, with the top-performing model achieving 68.1% weighted average accuracy.
- Scalable synthetic QA generation is a viable method for producing challenging, clinically grounded evaluation data at scale with minimal human oversight.
- Models exhibit varied performance across different reasoning types and audio characteristics, with some demonstrating resilience to synthetic audio artifacts and others showing specific weaknesses in processing physiological sounds or synthetic inputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.