Reproduce “LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Daniele Cipollone, Sergey Titov, Maliheh Izadi, Egor Bogomolov, Arie van Deursen
LibEvoBench evaluates models’ knowledge of library APIs across versions using API Calling, API Identification, and Signature Recall tasks, with compatibility ground truth for PyTorch, NumPy, and SciPy. Across 13 models, performance on evolving APIs declines toward newer releases while stable-API performance stays nearly flat. In the zero-thinking-budget, no-tools evaluation, version pins add little, while redacted API descriptions help substantially.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.