Reproduce “GDCNet: Generative Discrepancy Comparison Network for Multimodal Sarcasm Detection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shuguang Zhang, Junhong Lian, Guoxin Yu, Baoxun Xu, Xiang Ao
A framework for multimodal sarcasm detection, GDCNet, models cross-modal conflicts by having an MLLM objectively describe images and then quantifying the semantic and sentiment differences with the accompanying text. This approach set a new state-of-the-art on the MMSD2.0 benchmark, achieving 87.38% accuracy and 86.34% F1-score.
Key insights
- Utilizing MLLMs for objective image description generation, rather than subjective sarcastic content, provides a more reliable foundation for quantifying cross-modal incongruity in sarcasm detection.
- Explicitly modeling semantic and sentiment discrepancies between an objective image description and the original text is crucial for identifying nuanced ironic cues.
- An adaptive gating mechanism for multimodal feature fusion effectively balances the input from visual, textual, and discrepancy-based features, enhancing overall classification robustness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.