Reproduce “CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale” on OpenResearch

About this paper

Chenlong Wang, Zhaoyang Chu, Zhengxiang Cheng, Xuyi Yang, Kaiyue Qiu, Yao Wan, Zhou Zhao, Xuanhua Shi and 1 more

The research introduces CODESYNC, a scalable data engine for collecting authentic, real-time API evolution data from widely used Python libraries, and CODESYNCBENCH, a comprehensive benchmark to evaluate Large Language Models' (LLMs) ability to adapt to these dynamic changes. The study reveals that current LLMs exhibit significant knowledge obsolescence for evolving APIs, and existing knowledge updating methods offer limited effectiveness, particularly for complex method APIs.

Key insights

• State-of-the-art LLMs, including recent proprietary and open-source models, consistently exhibit poor performance on tasks involving dynamic API updates, often scoring below 20% BLEU on code completion.
• Current knowledge updating methods (e.g., SFT-LoRA, DPO) improve LLM performance on API evolution tasks but still yield low absolute scores, indicating their general insufficiency for effectively internalizing dynamic code knowledge.
• LLMs demonstrate significantly worse performance when adapting to updates in method APIs compared to function or initializer APIs, highlighting the increased complexity associated with object types and class hierarchies.

Summary by alphaXiv.