Reproduce “Panoptic Scene Graph Generation” on OpenResearch

About this paper

Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, Ziwei Liu

This paper introduces Panoptic Scene Graph Generation (PSG), a new task that grounds scene graphs in pixel-level panoptic segmentations rather than bounding boxes, incorporating both objects and amorphous 'stuff' regions. The authors create a high-quality PSG dataset and establish strong one-stage Transformer-based baselines that achieve state-of-the-art performance, demonstrating improved visual grounding and comprehensive scene understanding.

Key insights

• Integrating panoptic segmentation into scene graph generation provides pixel-level accuracy and comprehensive scene context, including amorphous 'stuff' regions.
• Carefully curated predicate dictionaries and rigorous annotation processes are crucial for creating high-quality datasets that overcome limitations of prior scene graph benchmarks.
• One-stage Transformer-based architectures can effectively perform end-to-end panoptic scene graph generation, often outperforming multi-stage methods, especially with sufficient training.

Summary by alphaXiv.