Reproduce “InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Siqi Ouyang, Xi Xu, Lei Li
InfiniSST introduces an approach for simultaneous translation of unbounded streaming speech, formulating the task as a multi-turn dialogue leveraging a large language model. It achieves a 0.5 to 1 second reduction in computation-aware latency and a Real-Time Factor less than half of previous baselines, while maintaining competitive translation quality on MuST-C benchmarks.
Key insights
- Formulating SST as a multi-turn dialogue with an LLM and efficient Key-Value (KV) cache management is effective for reducing computation-aware latency while maintaining translation quality.
- Training with robust segments, derived from continuous talks and adapted to handle non-linguistic sounds, is critical for enabling models to generalize to unbounded, real-world speech.
- Multi-latency augmentation during training enhances trajectory diversity and improves the model's quality-latency trade-off and generalization capabilities to various input delays.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.