Reproduce “CodeAlignBench: Assessing Code Generation Models on Developer-Preferred Code Adjustments” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Forough Mehralian, Ryan Shar, James R. Rae, Alireza Hashemi
Researchers at Apple Inc. developed CodeAlignBench, a multi-language benchmark grounded in a user study with developers, to evaluate large language models' adherence to specific, non-functional code adjustments in addition to functional correctness. The evaluation of ten leading models revealed a significant performance gap, with models performing better on follow-up instructions (up to 24% median difference) and structural code changes, indicating substantial room for improvement in aligning generated code with developer preferences.
Key insights
- Providing contextual history (initial generated code) in 'follow-up' tasks significantly improves LLM instruction adherence compared to 'predefined' tasks where instructions are part of the initial problem prompt.
- Models demonstrate higher proficiency in implementing 'structural' code changes (e.g., altering constructs or data structures) compared to 'semantic' or 'cosmetic' adjustments.
- Large Language Models can reliably act as judges for verifying instruction adherence, with Claude Sonnet 4 achieving 86.67% accuracy against human assessments for verification tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.