Reproduce “Whisper Has an Internal Word Aligner” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sung-Lin Yeh, Yen Meng, Hao Tang
Researchers from the University of Edinburgh found that OpenAI's Whisper model inherently contains precise word alignment information within its attention mechanisms. They developed an unsupervised, training-free method utilizing character-level teacher forcing and an L2 norm-based filter to extract highly accurate word-level timestamps, achieving substantial F1 score improvements over previous methods on various datasets.
Key insights
- OpenAI's Whisper model, without explicit training for alignment, contains attention heads capable of producing highly accurate word-level timestamps.
- Employing character-level tokenization during teacher-forced decoding yields significantly sharper and more monotonic attention maps, enabling finer-grained alignment extraction compared to wordpiece tokenization.
- The proposed L2 norm-based criterion successfully identifies specific attention heads that are specialized in capturing temporal alignments, indicating their interpretable roles within the model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.