Reproduce “Hierarchical Transformer-based Large-Context End-to-end ASR with Large-Context Knowledge Distillation” on OpenResearch

About this paper

Ryo Masumura, Naoki Makishima, Mana Ihori, Akihiko Takashima, Tomohiro Tanaka, Shota Orihashi

Researchers at NTT Media Intelligence Laboratories introduced a hierarchical Transformer for end-to-end automatic speech recognition, specifically designed for large contexts, achieving improved performance on Japanese discourse tasks when combined with a large-context knowledge distillation method.

Key insights

• The hierarchical Transformer architecture effectively captures long-range sequential contexts, outperforming both utterance-level Transformers and hierarchical RNNs for large-context ASR.
• The proposed hierarchical Transformer-based model demonstrates robustness to ASR errors in preceding utterances, showing negligible performance degradation when using ASR hypotheses versus ground-truth contexts.
• Large-context knowledge distillation successfully guides the ASR model to better utilize and exploit extensive contextual information by mimicking a powerful pre-trained large-context language model.

Summary by alphaXiv.