Reproduce “Investigating Human-Model Discrepancies in Speech Quality Assessment via Acoustic and Prosodic Perturbations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Masato Takagi, Masaya Kawamura, Reo Shimizu, Yuma Shirahata
This study systematically compared human and automatic MOS prediction model assessments of speech quality across acoustically degraded, prosodically erroneous, and speaker-varied samples. It found that while models generally align with human judgments on acoustic fidelity, they exhibit striking insensitivity to prosodic (pitch-accent) errors and demonstrate significant biases in evaluating speaker characteristics not present in human perception.
Key insights
- Most automatic MOS models successfully tracked changes in signal-level acoustic degradation, aligning well with human judgments, though training data composition played a more dominant role than SSL encoder architecture for this task.
- All tested MOS prediction models showed negligible sensitivity to perceptually significant prosodic (pitch-accent) errors, despite these errors causing human MOS to drop by over 1.8 points.
- Models demonstrated systematic biases regarding speaker characteristics, particularly a strong negative correlation with mean F0 (assigning higher scores to lower F0 speakers) that was largely absent in human ratings, and were insensitive to F0 variability and speaking rate that humans deemed important.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.