Reproduce “Robust Harmful Meme Detection under Missing Modalities via Shared Representation Learning” on OpenResearch

About this paper

Felix Breiteneder, Mohammad Belal, Muhammad Saad Saeed, Shahed Masoudian, Usman Naseem, Kulshrestha Juhi, Markus Schedl, Shah Nawaz

Researchers at Johannes Kepler University, Aalto University, University of Michigan-Flint, and Macquarie University developed a shared representation learning method for harmful meme detection that maintains performance even when textual information is missing or noisy. This approach projects visual and textual embeddings independently into a shared latent space, outperforming existing multimodal baselines under conditions of modal incompleteness.

Key insights

• Independently processing and projecting modalities into a shared latent space significantly improves robustness to missing data compared to traditional cross-modal fusion techniques.
• The shared parameterization across modalities allows the model to effectively leverage visual features when textual information is unreliable or completely absent.
• Focusing on robustness to missing modalities is crucial for bridging the gap between research benchmarks and the practical deployment challenges of harmful meme detection systems.

Summary by alphaXiv.