Reproduce “Segmentwise Pruning in Audio-Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Marcel Gibier, Raphaël Duroselle, Pierre Serrano, Olivier Boeffard, Jean-François Bonastre

Researchers at Inria Paris investigated token pruning strategies for Audio-Language Models (ALMs) to enhance computational efficiency and performance, proposing a Segmentwise Top-K method. This approach consistently matched or surpassed existing methods, allowing up to 75% token reduction with minimal performance degradation, while also achieving up to a 23% relative improvement in CIDEr score on AudioCaps for certain models.

Key insights

  • Not all tokens in an audio sequence carry unique or equally important information, and many highly attended tokens are redundant or temporally close, suggesting intelligent pruning could reduce load and potentially provide a cleaner signal.
  • Incorporating the inherent temporal dimension of audio into pruning strategies, such as segmentwise selection, is crucial for effectively reducing tokens while ensuring a diverse and representative capture of information across the entire audio sequence.
  • Pruning not only reduces computational load but can also act as a noise reduction or redundancy elimination mechanism, potentially leading to improved performance for downstream language models by presenting a more focused signal.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.