Reproduce “Large Scale Multi-Task Bayesian Optimization with Large Language Models” on OpenResearch

About this paper

Yimeng Zeng, Natalie Maus, Haydn Thomas Jones, Jeffrey Tao, Fangping Wan, Marcelo Der Torossian Torres, Cesar de la Fuente-Nunez, Ryan Marcus and 2 more

BOLT (Bayesian Optimization with LLM Transfer) integrates Large Language Models into multi-task Bayesian optimization to overcome traditional methods' scalability limitations. This framework demonstrates continuous performance improvements across thousands of tasks in database query optimization and antimicrobial peptide design, achieving faster convergence and superior few-shot capabilities.

Key insights

• LLMs can effectively learn from extensive Bayesian optimization trajectories to provide high-quality warm-starts for new tasks, overcoming the scalability constraints of conventional multi-task BO.
• A positive feedback loop between BO and LLMs enables continuous improvement, as improved LLMs generate better initializations, leading to faster BO convergence and more high-quality data for further LLM refinement.
• The integration of latent space Bayesian optimization, using pre-trained Variational Autoencoders, allows BOLT to effectively optimize structured inputs like SQL queries and amino acid sequences.

Summary by alphaXiv.