Reproduce “A Language-Informed Loss for Visual Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Suchir Agarwal, Keshigeyan Chandrasegaran, Ziyu Chen, Kyle Sargent, Juan Carlos Niebles, Li Fei-Fei, Jiajun Wu
LILIE trains an invertible transform to preserve information useful for predicting captions from corrupted image features, then uses distances in that learned space as a generator training loss. Unlike a lossy tokenizer, it retains image information and leaves generation in pixel space. On 256×256 GPIC images, Pixel+LILIE lowers gFD from 66.84 to 53.01 versus pixel-only training in the JiT-T2I-g evaluation setup.
Key insights
- Invertibility lets LILIE provide a language-shaped comparison space without using a compressed image representation as the generator’s output space.
- Meaningful captions matter: replacing them with random token sequences removed the observed improvement over pixel-only training.
- LILIE’s benefit increased with encoder capacity and training data; its feature loss also improved results when used without a pixel-loss term.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.