Reproduce “Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Brian Yan, Xuankai Chang, Antonios Anastasopoulos, Yuya Fujita, Shinji Watanabe
A method for speech-to-text translation (ST) employs hard parameter sharing across speech and text modalities by discretizing speech and up-sampling text. This approach consistently improved ST performance by up to +1.8 BLEU points when leveraging external machine translation data and pre-trained text models.
Key insights
- By discretizing speech into token-like units and appropriately adjusting text sequence lengths, both modalities can be processed through a single, unified model architecture.
- Hard parameter sharing enables more effective utilization of abundant machine translation (MT) data, simplifying the architecture by removing modality-specific modules and the need for cross-modal regularization.
- The approach facilitates stronger transfer learning from pre-trained text models to speech translation tasks, enhancing the benefits of joint training across modalities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.