Reproduce “Towards Active Synthetic Data Generation for Finetuning Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Samuel Kessler, Menglin Xia, Daniel Madrigal Diaz, Dongge Han, Helia Heshemi, Saravan Rajmohan, Victor Ruehle, Jordan T. Ash
This work introduces an iterative, closed-loop framework for generating synthetic data to finetune Small Language Models (SLMs), demonstrating that actively curating synthetic data based on the student model's state significantly improves data efficiency. It achieves comparable or superior SLM performance with 1.3x to 2x fewer samples compared to static generation, with uncertainty sampling proving to be the most effective and computationally inexpensive data selection strategy.
Key insights
- Synthetic data generation is 'steerable,' meaning that aggregated properties of the teacher-generated data tend to reflect the characteristics of the seed samples used, allowing active selection to guide the generation of more relevant training examples.
- Simple active learning strategies, specifically uncertainty sampling based on the student model's next-token prediction loss on its own generations, are highly effective and computationally more efficient than complex LLM-as-a-judge methods.
- Using the student model's own predictions to calculate loss or reward scores for data selection is as effective as, or more effective than, using ground-truth answers for these calculations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.