Reproduce “Sarcasm in Sight and Sound: Benchmarking and Expansion to Improve Multimodal Sarcasm Detection” on OpenResearch

About this paper

Swapnil Bhosale, Abhra Chaudhuri, Alex Lee Robert Williams, Divyank Tiwari, Anjan Dutta, Xiatian Zhu, Pushpak Bhattacharyya, Diptesh Kanojia

This research updated multimodal sarcasm detection benchmarks by leveraging advanced deep learning encoders for video, audio, and text, establishing new performance baselines. It also introduced and validated MUStARD++ Balanced, a dataset extension addressing class imbalance, which collectively yielded a 2.4% macro-F1 improvement on the extended dataset, reaching 0.736 macro-F1.

Key insights

• Modern pre-trained deep learning encoders provide substantially better feature representations for multimodal sarcasm detection than classical approaches, leading to improved performance.
• Targeted dataset expansion, addressing class imbalance and adding diversity, leads to more robust models and enhanced accuracy across different sarcasm types.
• Fine-tuning audio encoders on Speech Emotion Recognition (SER) tasks is beneficial for sarcasm detection, indicating the value of emotion-aware features in multimodal understanding.

Summary by alphaXiv.