Reproduce “OpenPSG: Open-set Panoptic Scene Graph Generation via Large Multimodal Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zijian Zhou, Zheng Zhu, Holger Caesar, Miaojing Shi
OpenPSG introduces a framework for open-set panoptic scene graph generation, enabling the identification of both novel object categories and, critically, novel relation categories by leveraging large multimodal models. The approach achieves state-of-the-art performance in both closed-set and open-set scenarios, with a 20x inference speedup for relation prediction through a selective query mechanism.
Key insights
- Large Multimodal Models (LMMs) can effectively perform open-set relation prediction, with 'Judgement Instruction' proving more robust and efficient than free-form generation.
- The proposed Relation Query Transformer (RelQ-Former) efficiently filters irrelevant subject-object pairs, achieving a 20x inference speedup for relation prediction with minimal performance impact.
- Custom attention mechanisms within the RelQ-Former enhance the extraction of visual features specific to object interactions, improving relation prediction accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.