Reproduce “Unified Visual Relationship Detection with Vision and Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Long Zhao, Liangzhe Yuan, Boqing Gong, Yin Cui, Florian Schroff, Ming-Hsuan Yang, Hartwig Adam, Ting Liu

Google Research introduces UniVRD, a framework that unifies visual relationship detection across multiple heterogeneous datasets by leveraging Vision and Language Models (VLMs). The model achieves a new state-of-the-art of 38.07 mAP on HICO-DET and performs comparably to or better than dataset-specific models across various benchmarks.

Key insights

  • Pre-trained Vision and Language Models (VLMs) can effectively unify semantically similar visual relationships across heterogeneous datasets via their aligned image and text embeddings.
  • A single unified VRD model can perform as well as, or even outperform, dataset-specific models, particularly demonstrating significant improvements on smaller datasets and scaling benefits on larger ones.
  • The combination of cascade training, Mosaics augmentation, and fine-tuning VLM components is crucial for achieving high performance and stability in unified visual relationship detection.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.