Reproduce “Evaluating Inference-Time Defenses Against Package Hallucination in LLM-Generated Code” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alberick Euraste Djire, Iyiola E. Olatunji, Melissa Tessa, Earl T. Barr, Jacques Klein, Tegawendé F. Bissyandé
The authors evaluate defenses against package hallucinations and correct registry checks by including standard-library modules. Across eight open-weight models and four languages, RAG has the lowest benign-prompt hallucination rate in 19 of 32 configurations, while Greedy has the strongest average utility–mitigation balance. Under adversarial prompts, RAG and Self-Refine lead in 29 of 32 configurations; findings are limited to synthetic tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.