Reproduce “Revisiting the Role of Natural Language Code Comments in Code Translation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Monika Gupta, Ajay Meena, Anamitra Roy Choudhury, Vijay Arya, Srikanta Bedathur

The Indian Institute of Technology, New Delhi and IBM Research - India study the utility of natural language code comments in improving LLM-based code translation. Their COMMENTRA approach, which iteratively adds comments to failed translations, demonstrates average performance gains of 6% to 51% with one iteration and up to 71% with cascaded models, showing that descriptive English comments placed near relevant code lines are most effective.

Key insights

  • Model-generated code comments can both significantly improve and sometimes harm LLM translation performance, depending on the specific LLM, language pair, and comment characteristics.
  • Comments with a 'descriptive' intent, focusing on 'what the code does,' are most beneficial for translation, while multi-intent or verbose author-written comments can introduce noise.
  • English was found to be the most effective language for comments, and fine-grained comments placed in close proximity to relevant code lines yielded superior benefits compared to external specifications or method-level comments.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.