Reproduce “Evaluating Speech-to-Text x LLM x Text-to-Speech Combinations for AI Interview Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rumi Allbert, Nima Yazdani, Ali Ansari, Aruj Mahajan, Amirhossein Afsharrad, Seyed Shahabeddin Mousavi
Researchers at micro1, in collaboration with USC and Stanford University, empirically evaluated five Speech-to-Text, Large Language Model, and Text-to-Speech combinations in a production AI interview system using over 5,000 real interviews. Their analysis revealed that Speech-to-Text quality most significantly impacts system performance, while highlighting a weak correlation between automated LLM-based quality metrics and subjective user satisfaction ratings.
Key insights
- Speech-to-Text (STT) component quality is the dominant factor influencing overall conversational AI system performance, with significant cascading effects on other components.
- There is a crucial disconnect and weak correlation between automated LLM-as-a-Judge objective quality metrics and subjective user satisfaction in voice-based AI interactions.
- Automated LLM-based evaluation is a scalable and consistent method for assessing technical and conversational quality but must be supplemented with direct human feedback to capture the full spectrum of user experience.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.