Reproduce “Robust Target Speaker Diarization and Separation via Augmented Speaker Embedding Sampling” on OpenResearch

About this paper

Md Asif Jalal, Luca Remaggi, Vasileios Moschopoulos, Thanasis Kotsiopoulos, Vandana Rajan, Karthikeyan Saravanan, Anastasis Drosou, Junho Heo and 2 more

A new method for enrollment-free target speaker diarization and separation significantly improves robustness by training with augmented noisy speaker embeddings and introducing an Overlapping Spectral Loss. This approach achieves a 71% relative improvement in Diarization Error Rate and a 69% relative improvement in concatenated minimum-permutation Word Error Rate on the LibriCSS dataset compared to prior state-of-the-art.

Key insights

• Explicitly training with augmented noisy and speaker-overlapped embeddings significantly improves a model's generalization capabilities for robust target speaker diarization and separation.
• An Overlapping Spectral Loss function can effectively enhance temporal coherence during separation, particularly in multi-speaker overlap regions, improving downstream ASR and diarization accuracy.
• A Voice Activity Detection (VAD) model pre-trained with noisy embeddings, while potentially showing worse performance on clean data, can lead to superior results in the subsequent, more complex speech separation task due to improved feature learning for robustness.

Summary by alphaXiv.