Reproduce “Length Aware Speech Translation for Video Dubbing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Harveen Singh Chadha, Aswin Shanmugam Subramanian, Vikas Joshi, Shubham Bansal, Jian Xue, Rupeshkumar Mehta, Jinyu Li
Microsoft researchers introduce Length-Sensitive Speech Translation (LSST) combined with Length-Aware Beam Search (LABS) to generate multiple translation candidates of varying lengths in a single decoding pass for video dubbing. This end-to-end solution improves temporal synchronization by 16.3% for Spanish-to-English and 19.9% for Korean-to-English, while maintaining competitive translation quality and introducing minimal computational overhead.
Key insights
- Conditioning an end-to-end speech translation model on predefined length categories (short, normal, long) allows the model to learn to generate translations of desired durations.
- Simultaneously exploring multiple length-conditioned paths within a single beam search (LABS) provides an efficient way to generate diverse length candidates for real-time applications.
- Using phoneme count ratios rather than character counts for length measurement offers a more linguistically consistent and universally applicable method for cross-lingual length control.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.