Reproduce “GRILL: Grounded Vision-language Pre-training via Aligning Text and Image Regions” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Woojeong Jin, Subhabrata Mukherjee, Yu Cheng, Yelong Shen, Weizhu Chen, Ahmed Hassan Awadallah, Damien Jose, Xiang Ren

GRILL introduces a pre-training approach that explicitly connects textual references to visual elements through novel hybrid sequences and object-word alignments. The model achieves strong zero- and few-shot generalization across diverse vision-language tasks, particularly excelling in grounding capabilities on benchmarks like Visual Question Answering and Flickr30k-entities.

Key insights

  • Using hybrid sequences during pre-training significantly enhances the model's grounding capabilities by explicitly linking textual references to image regions.
  • The combination of language modeling and discriminative objectives in pre-training leads to improved overall performance across tasks.
  • GRILL demonstrates strong zero-shot performance across multiple vision-language tasks, indicating effective general understanding without extensive fine-tuning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.