Reproduce “AI-Mediated Code Comment Improvement” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Maria Dhakal, Chia-Yi Su, Robert Wallace, Chris Fakhimi, Aakash Bansal, Toby Li, Yu Huang, Collin McMillan

Researchers at the University of Notre Dame empirically determined seven specific quality axes that influence programmers' perceptions of code comment quality. They developed an AI-mediated procedure that leverages combined AI and human feedback to fine-tune language models, enabling them to generate improved comments along these defined quality criteria. The approach showed that fine-tuned CodeLlama models achieved substantial improvements, with average BERTScore increases of 115% and USE increases of 30% across all quality axes.

Key insights

  • Programmers value code comments based on seven distinct quality axes: Logical, Precise, Unambiguous, Exhaustive (internalizing code details), and Troubleshooting, Contextualizing, Condensing (externalizing contextual information).
  • AI models can be effectively fine-tuned using a hybrid Reinforcement Learning from AI Feedback (RLAIF) and Human Feedback (RLHF) approach to generate comments that align with specific human-defined quality criteria, moving beyond simple imitation.
  • Smaller, in-house deployable models can achieve improvements in comment quality, demonstrating a practical approach for organizations concerned with data custody, though larger models show greater capacity for nuanced learning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.