Reproduce “ExPO-HM: Learning to Explain-then-Detect for Hateful Meme Detection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jingbiao Mei, Mingsheng Sun, Jinghong Chen, Pengda Qin, Yuhong Li, Da Chen, Bill Byrne
EXPO-HM, a system developed by researchers including those from the University of Cambridge, presents an explain-then-detect framework for hateful meme detection that achieves state-of-the-art results across binary classification, fine-grained classification, and reasoning quality. It notably surpasses direct detection models in accuracy while providing interpretable, policy-aligned explanations.
Key insights
- Structured policy manuals are more effective for LMM warmup than human-written explanations, significantly improving subsequent model performance across detection and explanation quality.
- A curriculum learning strategy that first solidifies fine-grained reasoning before integrating binary classification enhances the model's ability to produce detailed and policy-aligned rationales.
- Conditional Decision Entropy (CDE) serves as an effective reward signal, guiding models to generate explanations that align with decisive and accurate predictions, and correlates strongly with human and LLM judgments of reasoning quality.
- It is possible for Explain-then-Detect systems to achieve and even surpass the accuracy of direct detection models, demonstrating that interpretability need not come at the cost of performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.