Reproduce “NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Junkai Chen, Zhenhao Li, Xing Hu, Xin Xia

Researchers from Zhejiang University and York University developed NLPerturbator, an automated framework to study the robustness of code Large Language Models (LLMs) to real-world natural language variations in prompts. The study revealed that all six evaluated open-source code LLMs experienced performance degradation across 21 practitioner-informed perturbation types, with an average relative Pass@1 drop between 4.8% and 6.1% and up to 21.2% for specific cases.

Key insights

  • Software practitioners perceive specific natural language variations (e.g., synonym substitution, grammatical changes) as realistic, while largely rejecting random character-level perturbations commonly used in prior robustness research.
  • `NLPerturbator` effectively and naturally generates practitioner-informed perturbations, offering a valuable tool and benchmark for future LLM robustness studies.
  • All tested code LLMs lack sufficient robustness to real-world natural language variations, exhibiting consistent performance degradation across various perturbation types.
  • Certain perturbation categories, such as converting declarative to interrogative sentences, cause more significant performance drops, highlighting specific vulnerabilities in LLM understanding and response.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.