Reproduce “SciCodePile: A 128GB Corpus and Executable Benchmark for Challenging Scientific Code Generation” on OpenResearch

About this paper

Weifeng Sun, Ye Fan, Yuchen Chen, Gou Tan, Jieke Shi, Yuan Yidi, Swee Liang Wong, Jonathan Pan and 1 more

SCICODEPILE is introduced as a 128GB corpus of scientific code gathered from 37,737 public repositories, alongside an executable benchmark of 200 tasks featuring automated functional verification. This resource demonstrates that while current large language models struggle with scientific code generation, its data effectively improves model performance in both code completion and executable generation.

Key insights

• Generating functionally correct scientific code remains a highly challenging task for existing large language models, with even strong models achieving limited success rates.
• LLM failure patterns often include syntactic errors, incomplete implementations, missing file-level coherence, and difficulties with structural completeness and precise functional alignment to instructions.
• The SCICODEPILE corpus provides effective training signals, with pretraining and instruction tuning experiments on small models demonstrating substantial improvements in scientific code completion and executable code generation.

Summary by alphaXiv.