Reproduce “How to Train your Text-to-Image Model: Evaluating Design Choices for Synthetic Training Captions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Manuel Brack, Sudeep Katakol, Felix Friedrich, Patrick Schramowski, Hareesh Ravi, Kristian Kersting, Ajinkya Kale
Researchers at Adobe, hessian.AI, TU Darmstadt, and DFKI systematically investigated how different synthetic captioning strategies impact text-to-image model performance, revealing that randomizing caption lengths optimizes text alignment, aesthetic quality, and sample diversity simultaneously. The study confirmed that the distribution of key terms within training captions significantly influences output gender biases in generated images.
Key insights
- While stronger VLMs for caption generation improve text alignment, long, dense captions alone can compromise output aesthetics and sample diversity, particularly for minimal inference prompts, effectively pushing them out of distribution.
- Diversifying caption length during training, specifically by randomizing the lengths of synthetic captions (from minimal to long), is the most effective strategy to eliminate the trade-off between text alignment, output aesthetics, and sample diversity.
- The distribution of key terms within training captions strongly influences the output bias of the resulting T2I model, with gender distributions in generated images closely aligning with those in the training captions, highlighting captions' role in inherited biases.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.