Reproduce “Online Audio-Visual Autoregressive Speaker Extraction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zexu Pan, Wupeng Wang, Shengkui Zhao, Chong Zhang, Kun Zhou, Yukun Ma, Bin Ma
Researchers from Tongyi Lab, Alibaba Group, and National University of Singapore developed an online audio-visual autoregressive speaker extraction model that integrates a lightweight visual encoder and a novel autoregressive acoustic encoder. The system efficiently isolates target speech in real-time, achieving a 10.0 dB SI-SNRi on the LRS3 dataset while demonstrating robustness to dynamic changes in target speakers with significantly reduced computational resources.
Key insights
- Highly efficient visual processing is achievable with BlazeNet64, which delivers comparable speaker extraction performance to heavier ResNet18-based encoders but with significantly fewer parameters (0.1M vs 11.2M) and MACs (2.1B vs 12.9B).
- Leveraging frame-level past extracted speech through the autoregressive acoustic encoder provides a substantial performance gain (0.9 dB SI-SNRi) and enhances the model's ability to maintain temporal attention and adapt to target speaker changes.
- The proposed model demonstrates robust performance in "changing target" scenarios, maintaining consistent extraction quality before and after a speaker switch, which is crucial for practical multi-party communication systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.