Reproduce “Audio-Visual Representation Learning via Knowledge Distillation from Speech Foundation Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jing-Xuan Zhang, Genshun Wan, Jianqing Gao, Zhen-Hua Ling
A new audio-visual representation learning approach distills rich knowledge from large-scale speech foundation models into a joint audio-visual student model, improving robustness in noisy environments and performance in low-resource settings. The method shows superior results in tasks like VSR and AVSR, reducing Word Error Rates, and provides a stable, efficient pretraining process compared to prior self-supervised methods.
Key insights
- Distilling knowledge from large-scale, audio-only SFMs effectively overcomes the parallel data scarcity limitation for audio-visual representation learning.
- The use of 'soft labels' in KL divergence and applying distillation loss over all output regions enables richer and more stable knowledge transfer compared to discrete pseudo-labels or masked region-only approaches.
- Full face images provide more informative visual cues for speech tasks than just the lip region-of-interest, contributing to stronger visual speech recognition performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.