Reproduce “Efficient Data Selection at Scale via Influence Distillation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mahdi Nikdan, Vincent Cohen-Addad, Dan Alistarh, Vahab Mirrokni

Researchers from ISTA and Google Research developed Influence Distillation, a theoretically grounded framework for efficient data selection in large language model fine-tuning. This method enables performance comparable to full dataset training with significantly fewer samples, achieving up to 3.5x faster selection while using as little as 5% of the original data.

Key insights

  • Second-order influence information provides a robust mathematical basis for deriving optimal, target-aware sample weights for LLM fine-tuning, moving beyond heuristic approaches.
  • Efficient approximations, such as landmark-based gradient recovery and JVP embeddings, enable scaling influence-based methods to massive datasets while preserving performance gains.
  • For typical small learning rates in LLM fine-tuning, the second-order term in the influence objective is negligible, justifying the use of a computationally more efficient first-order approximation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.