Reproduce “Iterative Scene Graph Generation” on OpenResearch

About this paper

Siddhesh Khandelwal, Leonid Sigal

Researchers at the University of British Columbia and the Vector Institute for AI introduced an iterative, Transformer-based framework for Scene Graph Generation, reformulating the task as a multi-step refinement process to enable joint reasoning across subjects, objects, and predicates. The approach achieved state-of-the-art performance on Visual Genome and Action Genome, notably by introducing and excelling in Harmonic Recall, which balances performance across frequent and rare predicate classes.

Key insights

• Iterative refinement enables dynamic and joint reasoning among subjects, objects, and predicates, allowing mutual dependencies and mitigating error propagation throughout the graph generation process.
• A Transformer-based architecture can effectively implement this iterative refinement through explicit conditioning mechanisms that update predictions based on the full context of previous graph estimates.
• Loss re-weighting applied during training allows the model's internal feature representations to adapt to predicate class imbalance, offering a flexible trade-off between head and tail class performance.

Summary by alphaXiv.