Reproduce “MemeGraphs: Linking Memes to Knowledge Graphs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Vasiliki Kougia, Simon Fetzel, Thomas Kirchmair, Erion Çano, Sina Moayed Baharlou, Sahand Sharifzadeh, Benjamin Roth
MemeGraphs converts meme images and text into structured textual evidence by combining automatically generated scene graphs with Wikidata descriptions of named entities, then classifies offensive content using BERT. On the MultiOFF benchmark, the approach generally outperforms text-only and frozen-image baselines, reaching a best validation-selected F1 of 0.588, although adding all information sources does not consistently improve average performance and human correction offers no clear benefit.
Key insights
- Structured augmentations improve average classification over the text-only baseline: scene graphs reach F1 0.482 and Wikidata knowledge reaches 0.484, compared with 0.442 for TxtBERT and 0.408 for ImgBERT.
- The combined representation improves recall and produces the strongest selected checkpoint, but its average F1 is lower than either single augmentation, indicating that more context is not uniformly beneficial.
- Automatic and human-generated representations behave differently: automatic inputs are more effective when training and inference use the same noisy pipeline, while entity ambiguity, missed detections, and annotation disagreement limit the value of manual correction.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.