Reproduce “Language Models as Black-Box Optimizers for Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shihong Liu, Zhiqiu Lin, Samuel Yu, Ryan Lee, Tiffany Ling, Deepak Pathak, Deva Ramanan
Carnegie Mellon University researchers developed a framework where large language models (LLMs) act as black-box optimizers for vision-language models (VLMs), adapting them to downstream tasks using only natural language prompts and performance feedback. This approach improved 1-shot image classification accuracy by 5% over initial prompts, outperformed several white-box fine-tuning methods, and successfully refined prompts for text-to-image generation.
Key insights
- Large Language Models can effectively function as dynamic optimizers, inferring "implicit gradient" directions from conversational textual feedback to refine prompts.
- Optimizing within the natural language prompt space acts as a powerful regularizer, preventing overfitting and yielding robust performance in extremely low-shot scenarios.
- LLM-generated natural language prompts offer enhanced interpretability and exhibit superior black-box transferability across different VLM architectures compared to continuous prompts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.