Reproduce “SpeechPrune: Context-aware Token Pruning for Speech Information Retrieval” on OpenResearch

About this paper

Yueqian Lin, Yuzhe Fu, Jingyang Zhang, Yudong Liu, Jianyi Zhang, Jingwei Sun, Hai "Helen" Li, Yiran Chen

Researchers from Duke University introduce the Speech Information Retrieval (SIR) task and the SPIRAL benchmark, alongside SPEECHPRUNE, a training-free token pruning method. SPEECHPRUNE enables pre-trained Speech LLMs to process extended audio inputs more efficiently, achieving 89.23% accuracy on SPIRAL at a 20% pruning rate while reducing computational costs by 70%.

Key insights

• A combined two-phase pruning approach, leveraging both text-query relevance and approximated internal attention, is more effective at identifying and retaining crucial audio tokens for long-context understanding than either phase alone.
• Intelligent token pruning not only reduces computational overhead but can also enable pre-trained Speech LLMs to effectively address complex tasks where they previously showed no capability, such as the SPIRAL-H subset.
• SPEECHPRUNE demonstrates strong generalizability, improving performance across various speech understanding benchmarks and different Speech LLM architectures, indicating its robustness as a model-agnostic solution.

Summary by alphaXiv.