Reproduce “BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiaxing Wang, Deping Xiang, Jin Xu, Zirui Liu, Zicheng Zhang, Guoqiang Gong, Jun Fang, Chao Liu and 4 more
Researchers at JD.com and the University of Oxford developed BLADE, a scalable bi-level adaptive data selection framework for LLM training. It reformulates data selection to avoid computationally intensive Hessian calculations while dynamically updating a validation-informed reference model, leading to improved performance and efficiency over existing methods.
Key insights
- Principled bi-level optimization for data selection can be made computationally tractable for LLMs by transforming it into a penalized single-level problem, maintaining theoretical rigor without high overhead.
- The 'staleness' issue of excess-loss methods is effectively mitigated by dynamically updating a validation-informed reference model that tracks the proxy model's evolution.
- Efficient data scoring mechanisms (like excess-loss) can be derived from and integrated with a bi-level optimization framework, unifying previously distinct paradigms.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.