Reproduce “Implicit Inversion turns CLIP into a Decoder” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Antonio D'Orazio, Maria Rosaria Briglia, Donato Crisostomi, Dario Loi, Emanuele Rodolà, Iacopo Masi
This paper from Sapienza University of Rome demonstrates that the discriminative CLIP model can perform text-to-image synthesis without a dedicated generative decoder or fine-tuning, by implicitly inverting its vision encoder with Implicit Neural Representations. The approach yields improved image quality and semantic alignment compared to prior training-free inversion methods, achieving an FID of 72.5 on MS-COCO.
Key insights
- Discriminative models, such as CLIP, can possess inherent generative capabilities that can be unlocked through inversion without requiring dedicated generative decoders or additional training.
- Implicit Neural Representations (INRs), specifically FINER, can effectively represent and synthesize images when optimized against a frozen model, particularly when coupled with frequency-aware optimization strategies.
- Techniques like Adversarial Weight Perturbation and Orthogonal Procrustes analysis are critical for stabilizing the inversion process, bridging modality gaps, and guiding generation towards realistic and semantically aligned outputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.