Reproduce “Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Felix Wu, Kwangyoun Kim, Shinji Watanabe, Kyu Han, Ryan McDonald, Kilian Q. Weinberger, Yoav Artzi
Wav2Seq enables joint pre-training of speech-to-text encoder-decoder models using only raw audio by formulating a "pseudo ASR" task with automatically induced "pseudo languages." This approach improves performance in low-resource automatic speech recognition and speech-to-text translation, often outperforming encoder-only models.
Key insights
- Discrete "pseudo characters" derived from clustering speech representations can effectively mimic linguistic units for decoder pre-training.
- Deduplication and Byte-Pair Encoding (BPE) of these pseudo characters are crucial for creating compact, semantically relevant pseudo language sequences.
- Jointly pre-training the decoder with the encoder using a pseudo ASR task serves as an effective and computationally inexpensive second-stage pre-training step.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.