Reproduce “CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shuyan Zhou, Uri Alon, Sumit Agarwal, Graham Neubig
CodeBERTScore is a new automatic evaluation metric for natural-language-to-code generation that computes semantic similarity using contextualized embeddings from pretrained code models. This approach consistently achieves higher correlation with human judgments and functional correctness compared to previous token-matching and static analysis-based metrics, providing a practical alternative to execution-based evaluation.
Key insights
- Language-specific continued pretraining of CodeBERT models generally improves performance compared to base CodeBERT for code evaluation tasks.
- Utilizing contextual embeddings from deeper transformer layers (typically 7-11) often yields the highest correlation with human judgment and functional correctness.
- Incorporating the natural language instruction during the embedding generation process, even if its tokens are masked for final scoring, enhances the accuracy of code representations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.