Reproduce “GenMask: Adapting DiT for Segmentation via Direct Mask Generation” on OpenResearch

About this paper

Yuhuan Yang, Xianwei Zhuang, Yuxuan Cai, Chaofan Ma, Shuai Bai, Jiangchao Yao, Ya Zhang, Junyang Lin and 1 more

GenMask adapts the Diffusion Transformer (DiT) architecture to directly generate segmentation masks by framing segmentation as a conditional generation task, rather than a feature extraction problem. It introduces a tailored timestep sampling strategy for binary masks, achieving competitive performance on referring segmentation benchmarks (e.g., 83.3 oIoU on RefCOCO test A) and strong reasoning segmentation results (e.g., 52.3 gIoU on ReasonSeg test set).

Key insights

• Segmentation can be formulated as a direct conditional generation task within a DiT, removing the need for external decoders or complex feature extraction from generative models.
• Binary segmentation masks have distinct VAE latent properties compared to natural images, requiring a specific high-noise sampling strategy for effective learning within a unified framework.
• Injecting the VAE-encoded input image as a 'low-level shortcut' is essential for providing pixel-level detail, complementing the high-level semantics from the Vision-Language Model (VLM).

Summary by alphaXiv.