Reproduce “DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen-tau Yih, Daniel Fried and 2 more
DS-1000 introduces a benchmark for data science code generation, featuring 1000 problems from StackOverflow across seven Python libraries. It provides reliable multi-criteria execution-based evaluation and actively defends against memorization via problem perturbation, demonstrating that state-of-the-art models like Codex-002 achieve 43.3% accuracy, indicating substantial avenues for model improvement.
Key insights
- Real-world data science code generation demands models capable of handling natural language queries and producing idiomatic solutions across diverse libraries, a capability not fully captured by prior algorithmic benchmarks.
- Rigorous, multi-criteria evaluation combining functional correctness with checks for code quality (e.g., disallowing inefficient loops) is essential for truly reliable assessment of code generation models.
- Memorization remains a significant challenge for large language models on code generation tasks, necessitating proactive perturbation strategies in benchmarks to accurately measure generalization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.