Reproduce “Squeezeformer: An Efficient Transformer for Automatic Speech Recognition” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sehoon Kim, Amir Gholami, Albert Shaw, Nicholas Lee, Karttikeya Mangalam, Jitendra Malik, Michael W. Mahoney, Kurt Keutzer

An Automatic Speech Recognition (ASR) model named Squeezeformer re-engineers the Conformer architecture, introducing a Temporal U-Net and simplifying its components for improved efficiency and accuracy. Developed by researchers at the University of California, Berkeley, Squeezeformer achieved up to 1.40% lower Word Error Rate on LibriSpeech while reducing computational operations by as much as 40%.

Key insights

  • Temporal redundancy in deep ASR feature representations allows for effective temporal downsampling strategies, such as a U-Net architecture, to reduce computational overhead.
  • Architectural simplification, including unified activations and streamlined normalization (using learned scaling layers), can enhance efficiency and improve performance in ASR models.
  • Careful design choices at both macro- and micro-architectural levels, validated through systematic ablation studies, are crucial for balancing model complexity, efficiency, and accuracy.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.