Reproduce “DCP-Bench-Open: Evaluating LLMs for Constraint Modelling of Discrete Combinatorial Problems” on OpenResearch

About this paper

Kostis Michailidis, Dimos Tsouros, Tias Guns

Researchers from KU Leuven and the University of Western Macedonia introduce DCP-Bench-Open, a diverse benchmark for evaluating Large Language Models (LLMs) in constraint modelling of discrete combinatorial problems. The study demonstrates that Python-based frameworks (CPMpy, OR-Tools) generally yield higher LLM accuracy than domain-specific languages like MiniZinc, with advanced inference-time methods like repeated sampling and self-verification achieving up to 91% accuracy for gpt-5.1 on CPMpy.

Key insights

• LLMs exhibit higher accuracy when generating models for Python-based frameworks (CPMpy, OR-Tools) compared to the domain-specific language MiniZinc, likely due to their extensive pre-training on Python code.
• Inference-time compute methods, particularly the combination of repeated sampling and self-verification, substantially improve LLM accuracy in constraint modelling, by reducing detectable errors and enhancing logical correctness.
• LLMs frequently overfit to default data instances, demonstrating a performance decline when evaluated on multiple hidden instances, indicating a challenge in generalizing abstract problem constraints from specific examples.

Summary by alphaXiv.