Reproduce “Hypernymization of named entity-rich captions for grounding-based multi-modal pretraining” on OpenResearch

About this paper

Giacomo Nebbia, Adriana Kovashka

This research from the University of Pittsburgh addresses challenges posed by named entities in web-sourced image captions for multimodal learning by introducing hypernymization. Replacing specific entities with general categories improved model performance in grounding tasks and open-vocabulary detection, bridging the domain gap between diverse training data and standard evaluation benchmarks.

Key insights

• Named entities in web-scraped captions create a domain mismatch with named entity-free evaluation datasets, hindering multimodal model performance.
• Hypernymization effectively reduces this domain gap by replacing specific named entities with more general categories, making training data more aligned with evaluation benchmarks.
• Both knowledge-based (manual) and context-learned hypernymization approaches can improve model generalization for grounding and open-vocabulary detection tasks.

Summary by alphaXiv.