Reproduce “LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks” on OpenResearch

About this paper

Xin Zhou, Martin Weyssow, Ratnadira Widyasari, Ting Zhang, Junda He, Yunbo Lyu, Jianming Chang, Beiqi Zhang and 2 more

Researchers investigated data leakage in Large Language Models (LLMs) across 83 software engineering benchmarks, quantifying its prevalence, identifying contributing factors, and demonstrating its impact on performance evaluation. The study revealed varying leakage rates across different tasks and languages, and introduced LessLeak-Bench, a cleaned dataset to facilitate more reliable LLM assessments.

Key insights

• Data leakage prevalence varies significantly across SE benchmarks, with many showing negligible leakage but a subset exhibiting alarmingly high rates, such as QuixBugs (100%) and BigCloneBench (55.7%).
• High leakage is primarily caused by direct inclusion of benchmark repositories, overlap with projects used for benchmark creation, reliance on common coding platforms (e.g., LeetCode), and shared use of data sources like GitHub issues.
• Automated metrics like perplexity are not effective or reliable for detecting data leakage without access to the LLM's pre-training data, showing low accuracy (40-50%) in discriminating leaked from non-leaked samples.

Summary by alphaXiv.