Reproduce “RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yanli Wang, Yanlin Wang, Suiquan Wang, Daya Guo, Jiachi Chen, John Grundy, Xilin Liu, Yuchi Ma and 3 more

This work introduces RepoTransBench, a real-world multilingual benchmark designed for repository-level code translation, and RepoTransAgent, an iterative agent framework for this complex task. The study reveals that current large language models (LLMs) achieve a maximum success rate of 32.8% on holistic project translation, highlighting significant challenges, particularly when translating from dynamic to static programming languages.

Key insights

  • Translating code at the repository level presents a significant challenge for current LLMs, with even advanced agent-based methods achieving only a 32.8% success rate.
  • An inherent asymmetry in translation difficulty exists, where migrating from statically-typed to dynamically-typed languages is considerably easier (45-63% success rate) than the reverse (typically below 10%), indicating LLMs struggle with inferring explicit type information.
  • The effectiveness of LLM-based translation inversely correlates with repository complexity, highlighting the current models' limitations in handling extensive cross-file dependencies and larger codebases.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.