Reproduce “Disentangling the Roles of Representation and Selection in Data Pruning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yupei Du, Yingjin Song, Hugh Mee Wong, Daniil Ignatev, Albert Gatt, Dong Nguyen

This paper systematically disentangles data pruning into its core components: data representation and selection algorithm, empirically demonstrating that the choice of data representation is more critical for pruning effectiveness than the selection algorithm. The study finds that gradient-based and loss-based representations are consistently more effective than hidden-state-based representations for identifying impactful data points.

Key insights

  • Representations based on losses and gradients are inherently more informative and discriminative than raw hidden states, particularly for instances near the decision boundary.
  • The choice of data representation has a greater influence on the selected data subset than the choice of the selection algorithm, indicating strong representations are paramount.
  • Selection algorithms do not always align with their stated objectives (e.g., 'difficulty' algorithms might select easy instances), and there is no single universally best selection algorithm.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.