Reproduce “Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wendi Sang, Kai Li, Yifan Li, Jianqiang Huang, Xiaolin Hu
Falcon extracts a target speaker’s voice from mixed audio using mouth video, processing multiple time-frequency scales in one encoder–decoder pass. A training-only contrastive loss pulls target features toward the reference and away from near-field interferers. On RealSSA-Sim, Falcon reaches 7.7 dB SI-SNRi versus 7.6 dB for causal AV-TF-GridNet; this simulated-test result is not an end-to-end latency guarantee.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.