Reproduce “OPT-BENCH: Evaluating LLM Agent on Large-Scale Search Spaces Optimization Problems” on OpenResearch

About this paper

Xiaozhe Li, Jixuan Chen, Xinyu Fang, Shengyuan Ding, Haodong Duan, Qingwen Liu, Kai Chen

This paper presents OPT-BENCH, a new benchmark designed to evaluate Large Language Model agents on iterative optimization across large search spaces, encompassing 20 real-world machine learning challenges and 10 classical NP-hard problems. The research also introduces OPT-Agent, an iterative optimization framework that allows LLM agents to learn and refine solutions based on feedback, demonstrating that leveraging historical context significantly enhances optimization performance and that increasing iteration steps generally leads to better results for ML tasks.

Key insights

• Leveraging historical context consistently enhances optimization performance and reduces buggy rates for LLM agents.
• Increasing optimization steps generally improves performance for machine learning tasks, while gains for NP tasks can plateau after a certain number of iterations.
• Optimal temperature settings vary, with lower temperatures often leading to higher stability for proprietary models, while moderate temperatures can reduce buggy rates for NP tasks.

Summary by alphaXiv.