Reproduce “Rethinking Speech Representation Aggregation in Speech Enhancement: A Phonetic Mutual Information Perspective” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Seungu Han, Sungho Lee, Kyogu Lee
This work rethinks speech representation aggregation in speech enhancement, introducing a decoupled linguistic optimization approach where self-supervised learning feature aggregation modules are pre-trained to maximize phonetic mutual information. The methodology improves speech intelligibility, demonstrated by lower Word Error Rates, by prioritizing linguistic content and adapting aggregation dynamically to varying noise conditions.
Key insights
- Phonetic information in SSL models consistently peaks in upper layers (e.g., layers 9-11) even under noise, although its overall magnitude is significantly reduced compared to clean speech.
- Acoustic-tuned aggregation typically assigns higher weights to low-level acoustic features (Layer 0), while linguistic-tuned aggregation correctly prioritizes linguistically rich upper layers.
- Dynamic aggregation allows the model to adaptively shift focus, prioritizing top layers at high SNR and spreading weights to include middle layers at low SNR to effectively search for linguistic content.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.