Reproduce “ORGEval: Graph-Theoretic Evaluation of LLMs in Optimization Modeling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhuohan Wang, Ziwei Zhu, Ziniu Li, Congliang Chen, Yizhou Han, Yufeng Lin, Zhihang Lin, Angyang Gu and 3 more
The ORGEval framework provides a graph-theoretic evaluation method for Large Language Models (LLMs) in optimization modeling, achieving orders of magnitude faster evaluation and 100% consistency compared to traditional solver-based approaches. This method relies on representing optimization problems as bipartite graphs and testing for graph isomorphism under a novel 'symmetric decomposable' condition.
Key insights
- Evaluating optimization models through graph isomorphism testing offers a theoretically sound and empirically consistent measure of structural correctness.
- The 'symmetric decomposable' condition ensures the completeness of the Weisfeiler-Lehman test for detecting isomorphism in optimization problem graphs, overcoming a key limitation of the heuristic.
- Shifting from solver-based numerical outcomes to graph-theoretic structural comparison fundamentally improves evaluation reliability and efficiency for LLMs in OR.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.