Reproduce “PPL-Factory: Task-Aware and Budget-Aware Data Selection from Language Modeling to Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hang Zhang, Warren J. Gross
PPL-Factory selects fine-tuning examples using the target model’s next-token uncertainty, scoring reasoning steps and final answers separately for math tasks and adapting the selected score region to the data budget. For LLaMA3.2-3B-Instruct, a 10% subset yields 62.77% GSM8K exact match and 32.0% MATH Pass@1, above full-data results of 61.87% and 27.2% in these evaluations only.
Key insights
- The optimal data selection strategy for LLM fine-tuning is budget-dependent, transitioning from easier samples at high data ratios to more diverse, middle-range samples at low data ratios.
- Decomposing NLL scores for reasoning tasks into separate components for reasoning steps and final answers effectively identifies more informative examples than using a single NLL for the entire response.
- Introducing randomness into perplexity-based middle-band selection ('mid_random') is crucial for maintaining data diversity and improving performance when operating with extremely limited data budgets, without incurring additional computational cost.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.