Reproduce “FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anjia Cao, Xing Wei, Zhiheng Ma
FLAME is a framework that leverages frozen Large Language Models as text encoders for data-efficient language-image pre-training, significantly enhancing long-context understanding and achieving inherent multilingual generalization. The system attains state-of-the-art zero-shot retrieval and classification performance with less training data, outperforming baselines and dedicated multilingual models.
Key insights
- Frozen, powerful Large Language Models can effectively serve as text encoders in language-image pre-training, providing rich semantic context without requiring fine-tuning.
- Decomposing visual semantics into distinct facets through prompt engineering and leveraging the LLM's comprehension abilities leads to more comprehensive visual-semantic alignments.
- Combining facet-decoupled attention with an offline embedding strategy makes incorporating large, frozen LLMs computationally efficient and practical for large-scale pre-training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.