Reproduce “CODECLEANER: Elevating Standards with A Robust Data Contamination Mitigation Toolkit” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jialun Cao, Songqiang Chen, Wuqi Zhang, Hau Ching Lo, Shing-Chi Cheung
CODECLEANER is an open-source toolkit developed by HKUST researchers that systematically investigates and provides automated code refactoring operators to mitigate data contamination in Code Language Models. The toolkit demonstrates that applying all method-level operators can reduce N-gram overlap by 65%, providing a practical solution for more credible CLM evaluations.
Key insights
- Semantic refactoring operators like Performance Measurement Decoration (Deco) and Identifier Renaming (Renm) are most effective at reducing CLM familiarity with contaminated code.
- While syntactic operators reduce data-wise overlap, some can surprisingly increase CLM familiarity, suggesting their effects are not always beneficial for model-wise contamination.
- The severity of data contamination is generally similar across different programming languages, though mitigation efficacy can vary.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.