Reproduce “Code Execution with Pre-trained Language Models” on OpenResearch

About this paper

Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan, Nan Duan

This work introduces CodeExecutor, a Transformer-based model from Peking University and Microsoft that predicts detailed execution traces for Python programs, including executed lines and variable state changes. Utilizing a mutation-based data augmentation and curriculum learning strategy, the model achieves an output accuracy of 48.06% on competitive programming problems and enhances zero-shot code-to-code search MAP by 32.86 points.

Key insights

• Pre-training models with explicit code execution traces, capturing both control flow and variable state changes, provides a deeper semantic understanding of code than static analysis alone.
• Mutation-based data augmentation offers an effective and scalable method to generate diverse and realistic executable code-trace pairs, addressing the scarcity of such datasets.
• A curriculum learning approach is crucial for successfully training models on code execution, enabling them to gradually master increasing complexity and retain knowledge across different program types.

Summary by alphaXiv.