Reproduce “Coarse-Tuning Models of Code with Reinforcement Learning Feedback” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Abhinav Jain, Chima Adiole, Swarat Chaudhuri, Thomas Reps, Chris Jermaine
The paper introduces Coarse-Tuning Models of Code with Reinforcement Learning Feedback (RLCF), a method for enhancing large language models (LLMs) to generate functionally correct and syntactically valid code. RLCF employs a novel grounding function that combines compiler feedback for static semantic correctness with an LLM-based discriminator for "naturalness," leading to substantial reductions in compilation errors and functional failures, and enabling smaller LLMs to achieve performance comparable to much larger models.
Key insights
- A combined feedback mechanism, merging formal language tools like compilers with an adaptive LLM discriminator, effectively delivers dense and structured rewards for training code-generating LLMs without relying on costly unit tests or human annotations.
- Precisely localizing and penalizing compilation errors at the token level, along with detecting "bad practices" such as unused variables, is crucial for effective credit assignment in RL and for guiding the agent towards generating robust, idiomatic code.
- "Coarse-tuning" with this coordinated feedback enables smaller LLMs (e.g., CodeGen 350M, CodeT5 770M) to achieve code generation quality comparable to significantly larger, un-tuned models, indicating a potential for more efficient and accessible AI-powered code generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.