Reproduce “Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Li Zhong, Zilong Wang
Researchers at the University of California, San Diego developed ROBUSTAPI, a benchmark that assesses the reliability of Large Language Model (LLM) generated code by detecting common Java API misuses. Their evaluation revealed high rates of API misuse in major LLMs, including GPT-4, though providing correct API usage examples significantly reduced these errors for some models.
Key insights
- API misuse is a pervasive issue across all evaluated LLMs, even those perceived as advanced, indicating that functional correctness does not equate to robust and reliable code generation.
- Providing concrete examples of correct API usage within the prompt (one-shot-relevant) significantly improves the reliability of generated code for leading LLMs like GPT-3.5 and GPT-4.
- LLMs learn API best practices more effectively from specific code demonstrations than from abstract natural language rules, challenging assumptions about how models absorb engineering principles.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.