Reproduce “FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lihe Yang, Xiaogang Xu, Bingyi Kang, Yinghuan Shi, Hengshuang Zhao

FreeMask, developed by researchers from HKU, Zhejiang Lab, ByteDance, and Nanjing University, is a framework leveraging mask-conditioned diffusion models to generate densely annotated synthetic images. It demonstrates that these processed synthetic images can perform comparably to real data for semantic segmentation and consistently boost state-of-the-art architectures by up to 3.3% mIoU on ADE20K.

Key insights

  • Sophisticated pixel-level filtering and hardness-aware re-sampling are crucial for synthetic data from generative models to effectively augment training and prevent performance degradation in dense prediction tasks.
  • Well-processed synthetic images generated by mask-conditioned diffusion models can achieve performance comparable to (within -0.2% mIoU for SegFormer-B4 on ADE20K) models trained exclusively on real data, showcasing their potential to reduce reliance on costly real annotations.
  • By strategically synthesizing and integrating 'harder' samples, the FreeMask framework allows for scaling data volume and diversity, leading to consistent performance boosts on already highly optimized state-of-the-art semantic segmentation models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.