Reproduce “Evaluating Language Models as Synthetic Data Generators” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Seungone Kim, Juyoung Suk, Xiang Yue, Vijay Viswanathan, Seongyun Lee, Yizhong Wang, Kiril Gashteovski, Carolin Lawrence and 2 more
Researchers at Carnegie Mellon University and collaborators introduced AGORABENCH, a unified benchmark for systematically evaluating large language models as synthetic data generators. The study found that GPT-4o consistently performs best in instance generation, while Claude-3.5-Sonnet excels in quality enhancement, and importantly, an LM's general problem-solving ability does not directly correlate with its effectiveness in generating training data.
Key insights
- An LM's general problem-solving ability does not strongly correlate with its effectiveness as a synthetic data generator, indicating distinct 'solver' and 'teacher' capabilities.
- Intrinsic features of generated data, such as response quality, instruction difficulty, and diversity, collectively predict data generation ability more effectively than problem-solving scores alone.
- Careful meta-prompt engineering is crucial for data generation quality, with optimized free-form prompts significantly outperforming unoptimized or structured JSON-format prompts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.