Reproduce “AmorLIP: Efficient Language-Image Pretraining via Amortization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haotian Sun, Yitong Li, Yuchen Zhuang, Niao He, Hanjun Dai, Bo Dai
AmorLIP reduces repeated CLIP normalization work by learning lightweight predictors of the partition function from image and text embeddings, then reusing estimates across batches. In large-scale ViT-B/32 training on CC-12M, its l2-log variant reaches 30.66% average performance across 38 DataComp tasks, versus CLIP’s 27.65%, within evaluations limited to ViT-B/32 and about 8.06 million examples.
Key insights
- CLIP's contrastive objective can be effectively and insightfully reformulated as an Energy-Based Model, highlighting the partition function as the key source of computational intractability.
- The partition function is linearly representable by basis functions, theoretically justifying its efficient approximation using simple Multi-Layer Perceptrons (MLPs).
- Parametric amortization provides a more scalable and efficient alternative to memory-buffer-based methods for managing negative samples, crucial for training on web-scale datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.