Reproduce “KNN-Diffusion: Image Generation via Large-Scale Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shelly Sheynin, Oron Ashual, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, Yaniv Taigman
Researchers at Meta AI developed kNN-Diffusion, a multi-modal AI framework that uses large-scale retrieval to achieve high-quality text-to-image generation from image-only datasets. This approach yields lower FID scores on benchmarks like MS-COCO (12.5) compared to previous textless training methods and enables mask-free image manipulation and out-of-distribution generation.
Key insights
- Large-scale retrieval of relevant image embeddings can effectively compensate for the absence of paired text-image training data, allowing for high-quality text-to-image generation with image-only datasets.
- Offloading 'world knowledge' to an external, non-trainable retrieval database enables significantly smaller generative models (e.g., 400M parameters) to achieve performance comparable to much larger models trained on paired data.
- Dynamically swapping the retrieval database at inference time allows a single trained model to adapt and generate images consistent with new styles or distributions (out-of-distribution generation) without requiring retraining.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.