Reproduce “Cross-attention and Self-attention for Audio-visual Speaker Diarization in MISP-Meeting Challenge” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhaoyang Li, Haodong Zhou, Longjie Luo, Xiaoxiao Li, Yongxin Chen, Lin Li, Qingyang Hong
An audio-visual diarization system for the MISP 2025 Meeting Challenge replaces simple feature concatenation with bidirectional cross-attention and temporal self-attention, while adding x-vector speaker representations, pseudo-label refinement, augmentation, and post-processing. The full system reaches 8.18% DER on the evaluation set versus 15.52% for the official baseline, a 47.3% relative reduction.
Key insights
- Attention-based multimodal fusion is the dominant contributor: removing CASA-Net raises evaluation DER from 8.18% to 17.04%, worse than the official baseline.
- Both modalities can complement one another under imperfect alignment, while temporal self-attention helps use conversational context beyond individual frames.
- Supporting techniques also matter: removing pseudo-label refinement, ECAPA-TDNN embeddings, mixup, or post-processing increases evaluation DER to 10.41%, 11.00%, 9.71%, and 9.03%, respectively.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.