Reproduce “Decoding Data Quality via Synthetic Corruptions: Embedding-guided Pruning of Code Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yu Yang, Aaditya K. Singh, Mostafa Elhoushi, Anas Mahmoud, Kushal Tirumala, Fabian Gloeckle, Baptiste Rozière, Carole-Jean Wu and 2 more
SCIP uses synthetic corruption to identify embedding signals for pruning code files. The authors compare original and corrupted examples, then rank files by cluster size and centroid distance. With 20% pruned, a fine-tuned 1.5B LLaMA raises HumanEval pass@1 from 25.0% to 28.0%, but scores 33.0% on MBPP versus 33.4% without pruning, in this setup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.