Reproduce “M-BEST-RQ: A Multi-Channel Speech Foundation Model for Smart Glasses” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yufeng Yang, Desh Raj, Ju Lin, Niko Moritz, Junteng Jia, Gil Keren, Egor Lakomkin, Yiteng Huang and 3 more

Researchers at Meta developed M-BEST-RQ, the first multi-channel speech foundation model for smart glasses, employing a geometry-agnostic design with fixed beamformers and self-supervised learning. This model surpassed supervised baselines on conversational ASR and achieved state-of-the-art performance on spherical active source localization using only audio input, demonstrating strong generalizability across diverse tasks and device configurations.

Key insights

  • Large-scale self-supervised pre-training on synthetic data can significantly reduce the need for extensive labeled data, enabling strong performance even with limited fine-tuning data.
  • Combining both simulated and real multi-channel data during pre-training yields synergistic benefits, particularly when the volume of simulated data is moderate.
  • The fixed beamformer approach effectively renders the subsequent neural encoder agnostic to the specific microphone array geometry, allowing cross-device generalization between different smart glasses hardware.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.