Reproduce “IndustryCode: A Benchmark for Industry Code Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Puyu Zeng, Zhaoxi Wang, Zhixu Duan, Liang Feng, Shaobo Wang, Cunxiang Wang, Jinghang Wang, Bing Zhao and 2 more
Researchers introduce IndustryCode, a new benchmark to evaluate Large Language Models (LLMs) in industrial code generation, comprising 125 main challenges decomposed into 579 sub-problems across four programming languages and twenty domains. Evaluations show state-of-the-art LLMs struggle with long-horizon tasks, domain-specific knowledge, and niche languages, with Claude 4.5 Opus achieving the highest overall Pass@1 accuracy of 68.1% on sub-problems.
Key insights
- A significant 'Execution Gap' exists between sub-problem and main problem accuracy, indicating LLMs struggle with orchestrating multiple steps and maintaining logical coherence over long-horizon, system-level tasks.
- Model failures are predominantly due to syntactic deficits (32.8%) and semantic misalignment (30.2%) rather than fundamental reasoning errors (8.4%), suggesting a lack of specialized vertical-domain corpora.
- Activating 'thinking mode' can improve accuracy in complex derivations but may introduce 'Reasoning-Induced Errors' like cognitive over-correction and contextual interference.
- Performance is strongly correlated with language popularity and domain specificity, with proprietary and niche languages like Stata and MATLAB showing significantly lower accuracy due to imbalanced training data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.