Reproduce “Show, Interpret and Tell: Entity-aware Contextualised Image Captioning in Wikipedia” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Khanh Nguyen, Ali Furkan Biten, Andres Mafla, Lluis Gomez, Dimosthenis Karatzas
This research introduces Wikipedia Captioning, a new task that requires models to generate image captions by integrating visual content with structured textual context from Wikipedia articles. The study proposes Masked Named Entity Modeling (MNEM), a pre-training objective that improves the generation of specific Named Entities, leading to state-of-the-art performance with transformer-based models like T5 and demonstrating positive knowledge transfer to news captioning.
Key insights
- Transformer models significantly outperform traditional sequence-to-sequence approaches for Wikipedia captioning, with image input crucially improving performance.
- Models exhibit a 'copying shortcut' behavior, over-relying on textual overlap from the input description rather than fully integrating visual information, particularly evident in 'hard' samples.
- While Masked Named Entity Modeling (MNEM) improves factual grounding, current models still struggle with human-like 'context-based story-telling' for nuanced or mismatched scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.