Reproduce “Beyond Generation: Harnessing Text to Image Models for Object Detection and Segmentation” on OpenResearch

About this paper

Yunhao Ge, Jiashu Xu, Brian Nlong Zhao, Neel Joshi, Laurent Itti, Vibhav Vineet

A novel framework harnesses the capabilities of Text-to-Image models to automatically generate large-scale, pseudo-labeled datasets for object detection and instance segmentation. This approach achieves performance comparable to models trained on full real datasets and provides substantial improvements when combined with limited real data, addressing the challenge of data scarcity and annotation costs.

Key insights

• Large Text-to-Image models can be effectively leveraged to generate accurately labeled data for discriminative computer vision tasks, moving beyond their primary generative function.
• Decoupling foreground object synthesis (with mask extraction) from language-driven contextual background synthesis enables the creation of high-quality, compositionally controlled synthetic training data.
• Synthetic data generated via this method can significantly complement real-world data, leading to superior performance for object detection and instance segmentation, particularly in low-resource settings.

Summary by alphaXiv.