Reproduce “Occlusion-Aware Panoptic Segmentation with Joint Position Embedding and Occlusion-Level Attention” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wenbo Wei, Jun Wang, Shan Raza, Abhir Bhalerao

Researchers from the University of Warwick developed PEMOLA, a lightweight module that enhances transformer-based panoptic segmentation by integrating image-level occlusion priors into position embeddings. This method improved Panoptic Quality (PQ) by +0.8 for Mask2Former on both COCO-OLAC and Cityscapes-OLAC datasets, and by +0.8 for Mask DINO on COCO-OLAC.

Key insights

  • Dynamically adapting position embeddings with image-level occlusion priors significantly enhances panoptic segmentation robustness in occluded scenes, without requiring instance-level amodal annotations.
  • Both spatial occlusion-level attention and channel-wise occlusion label embedding contribute synergistically to the module's effectiveness.
  • Modulating position embeddings specifically within the pixel decoder, rather than the transformer decoder, is crucial for effectively leveraging occlusion cues, suggesting better compatibility with lower-level spatial features.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.