Reproduce “Frame-Aligned Fusion of Canary and WavLM for Non-Intrusive Intelligibility Prediction of Hearing-Aid-Processed Speech” on OpenResearch

About this paper

Kazushi Nakazawa

Kazushi Nakazawa at Advanced Media, Inc. introduced a frame-aligned fusion method combining Canary and WavLM speech encoders for non-intrusive intelligibility prediction of hearing-aid-processed speech. This approach, which prepares fine-grained WavLM features to align with Canary's coarser timeline, achieved an Eval RMSE of 24.96±0.06 and an Eval Corr of 0.796±0.001, outperforming single-backbone baselines and other fusion strategies.

Key insights

• Frame-aligned feature fusion before utterance-level pooling, particularly mapping finer acoustic features (WavLM) to a coarser linguistic reference (Canary), yields superior intelligibility prediction.
• The benefit of fusion stems from a beneficial inductive bias from the interaction strategy, rather than simply increased model capacity, as the best model had fewer trainable parameters.
• While strict temporal synchrony is not essential, maintaining coarse local temporal correspondence between features is important for the observed performance gains.

Summary by alphaXiv.