Reproduce “ITI-GEN: Inclusive Text-to-Image Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Cheng Zhang, Xuanbai Chen, Siqi Chai, Chen Henry Wu, Dmitry Lagun, Thabo Beeler, Fernando De la Torre
ITI-GEN learns category-specific prompt embeddings from reference images, keeping the text-to-image model frozen. It combines these embeddings with prompts and samples category combinations in chosen proportions. On a four-attribute test, its distribution discrepancy was 0.094, versus 0.476 for hard-prompt search and 1.406 for Stable Diffusion. Results demonstrate control over selected attributes, but performance weakens for subtle or entangled combinations.
Key insights
- Images can represent nuanced attributes more expressively and unambiguously than text, making them effective guides for learning inclusive text embeddings.
- Learning compact, modular prompt embeddings while keeping the large generative model frozen provides an efficient and scalable approach to debiasing.
- Aligning the directional differences in visual and textual embedding spaces effectively teaches the model to disentangle and control specific attributes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.