Reproduce “Retrieval Augmented Enhanced Dual Co-Attention Framework for Target Aware Multimodal Bengali Hateful Meme Detection” on OpenResearch

About this paper

Raihan Tanvir, Md. Golam Rabiul Alam

Researchers from BRAC University and Ahsanullah University of Science and Technology developed an Enhanced Dual Co-attention framework (xDORA) and retrieval-augmented strategies for detecting hateful multimodal Bengali memes. The methodology achieved a macro-averaged F1-score of 0.79 for hateful meme identification and 0.74 for target entity detection, demonstrating improved performance over existing models and specific strength in classifying rare categories.

Key insights

• Integrating powerful, pre-trained vision and multilingual text encoders with dual co-attention significantly improves multimodal fusion for understanding complex hateful memes in low-resource languages.
• Retrieval-augmented approaches, such as RAG-Fused DORA, provide substantial performance gains by combining supervised predictions with contextual knowledge from semantically similar samples.
• Non-parametric methods like FAISS-Based k-NN are particularly effective for accurately classifying rare and underrepresented classes, leveraging semantic proximity in embedding space to overcome data imbalance.

Summary by alphaXiv.