Reproduce “Prompt Amplification and Zero-Shot Late Fusion in Audio-Language Models for Speech Emotion Recognition” on OpenResearch

About this paper

Saurabh Kataria, Xiao Hu

Emory University researchers developed ZS-Fuse, a late-fusion method enhancing Speech Emotion Recognition by combining zero-shot predictions from Audio-Language Models (ALMs) with fine-tuned Foundation Models. This method, featuring a novel prompt amplification technique, achieved up to 0.8984 UAR on RAVDESS, surpassing existing state-of-the-art baselines.

Key insights

• Dual-encoder Audio-Language Models, when effectively prompted, are "underutilized" and can achieve strong zero-shot performance in SER, sometimes surpassing supervised baselines (e.g., CLSP's 0.742 UAR on RAVDESS).
• Prompt amplification can significantly enhance ALM zero-shot prediction capabilities and, when integrated into ZS-Fuse, leads to substantial improvements in SER Unweighted Average Recall (UAR).
• The optimal configuration for prompt amplification (audio and text repetitions) is highly variable and requires careful discrete optimization, as sub-optimal settings can degrade overall performance.

Summary by alphaXiv.