Reproduce “Speech Vecalign: an Embedding-based Method for Aligning Parallel Speech Documents” on OpenResearch

About this paper

Chutong Meng, Philipp Koehn

Speech Vecalign introduces an embedding-based method for aligning parallel speech documents, adapting a dynamic programming approach to leverage chronological order and document pairing. The method consistently improved Speech-to-Speech Translation (S2ST) model performance, for instance, achieving 12.58 ASR-BLEU for En-to-De and 16.14 ASR-BLEU for De-to-En on the EPST test set, while also generating longer, more context-rich alignments using less raw data than previous approaches.

Key insights

• High-quality speech-to-speech alignments predominantly occur within paired documents and maintain chronological order, affirming the applicability of sequence-aware alignment methods to speech.
• Explicitly enforcing chronological order and restricting alignment search to document pairs through dynamic programming effectively reduces noise and allows for the generation of longer, more coherent aligned segments.
• The accurate detection and removal of identical untranslated segments are critical for S2ST model performance, as multilingual speech embeddings can assign high similarity to such content regardless of language.

Summary by alphaXiv.