Reproduce “Beyond BLEU: A Semantic Evaluation Method for Code Translation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Julius Näumann, Sven Keidel, Amir Molzam Sharifloo, Mira Mezini
Introducing a semantic evaluation framework for code translation, researchers at TU Darmstadt demonstrated that Meta's LLMCompiler achieved semantic correctness scores of 0.33 and 0.63 for O0 and O3 optimized binaries respectively, while revealing traditional BLEU metrics poorly correlate with actual functional accuracy.
Key insights
- Traditional syntactic similarity metrics (BLEU, CodeBLEU) exhibit very weak correlation (point-biserial correlation coefficients from -0.13 to 0.35) with the actual semantic correctness of translated code.
- Semantically correct translations can have low syntactic similarity, and conversely, high syntactic similarity does not guarantee semantic preservation.
- LLM-based decompilers, exemplified by LLMCompiler, can achieve substantially higher semantic correctness scores compared to heuristic-based decompilers when evaluated using a rigorous semantic method.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.