Reproduce “SGTR+: End-to-end Scene Graph Generation with Transformer” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rongjie Li, Songyang Zhang, Xuming He

SGTR+ presents an end-to-end Transformer-based framework that formulates Scene Graph Generation as a bipartite graph construction problem. This approach jointly generates entity and predicate nodes, achieving state-of-the-art performance on OpenImages V6 (e.g., wmAP_rel of 39.54), Visual Genome, and GQA, with improved inference efficiency.

Key insights

  • Scene Graph Generation can be effectively modeled as an end-to-end bipartite graph construction problem, allowing for joint optimization of object detection and relationship prediction.
  • Incorporating spatial cues directly into predicate queries and performing ROI-Align on image-level predicate features enhances the efficiency and quality of predicate node generation.
  • Replacing heuristic triplet selection with a learnable, differentiable soft prediction module for graph assembling improves training stability and overall performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.