Reproduce “M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiaheng Liu, Ken Deng, Congnan Liu, Jian Yang, Shukai Liu, He Zhu, Peng Zhao, Linzheng Chai and 8 more
M²RC-EVAL introduces a massively multilingual, repository-level code completion benchmark covering 18 programming languages with fine-grained AST-based annotations. This work also provides M²RC-INSTRUCT, a high-quality instruction corpus that significantly improves code Large Language Models' performance and cross-lingual transferability for complex code generation tasks.
Key insights
- Repository-level context through retrieval and fine-tuning on targeted instruction data are crucial for boosting code LLM performance beyond in-file context alone.
- Code LLMs exhibit significant performance disparities across different programming languages and struggle more with higher-level, more abstract code structures (shallower AST nodes).
- Fine-tuning on a multilingual instruction corpus (M²RC-INSTRUCT) can yield strong cross-lingual transferability of instruction-following capabilities, improving performance across all included languages even when trained on a single language.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.