Reproduce “DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zeeshan Hayder, Xuming He

Researchers from Data61-CSIRO and ShanghaiTech University developed DSGG, an end-to-end Transformer-based framework that directly predicts scene graphs by employing novel graph-aware queries. The method achieved state-of-the-art performance on both Scene Graph Generation (Visual Genome) and Panoptic Scene Graph Generation (PSG) datasets, particularly excelling in handling multi-label relationships and low-frequency predicates while being more parameter-efficient.

Key insights

  • Formulating scene graph generation as a direct graph prediction problem with 'graph-aware queries' allows for efficient and holistic learning of scene structures.
  • Utilizing Sigmoid activation for multi-label relation prediction enables models to accurately capture multiple simultaneous relationships between object pairs, addressing relational semantic overlap.
  • A relaxed sub-graph matching strategy and relation re-scoring/distillation during training effectively mitigate challenges posed by long-tailed relation distributions and noisy annotations, improving performance on low-frequency relations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.