Reproduce “Towards Better Correctness and Efficiency in Code Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yunlong Feng, Yang Xu, Xiao Xu, Binyuan Hui, Junyang Lin
Researchers from Alibaba Group developed a two-stage tuning strategy to enhance the correctness and runtime efficiency of code generated by large language models. Their method, which combines offline DPO for initial correctness with online RLOO for subsequent efficiency optimization using a novel CPU instruction-based reward, improved average code correctness by 10.18% and efficiency by 7.75% on a 7B model, achieving performance comparable to larger 32B models.
Key insights
- Offline fine-tuning methods are inherently limited by the quality of the initial candidate solutions from the base model, struggling to improve efficiency for complex problems if the base model's correct solutions are already inefficient.
- A direct trade-off exists between correctness and efficiency during training; an exclusive focus on efficiency without considering correctness can degrade overall performance.
- Initializing online reinforcement learning with a highly accurate DPO-tuned model is crucial for stable efficiency optimization, and error-insensitive algorithms like RLOO are more effective at maintaining correctness during efficiency-focused training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.