Reproduce “Efficacy of Synthetic Data as a Benchmark” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gaurav Maheshwari, Dmitry Ivanov, Kevin El Haddad
The authors compare real and LLM-generated test data across six models, six datasets, and three NLP tasks. Ranking agreement can exceed score agreement: SNIPS macro-F1 averages MSPD 0.00 and SRCC 0.94, while ATIS averages 0.06 and 0.94. Agreement varies by task, particularly for NER, so synthetic data cannot be assumed to predict real-world scores.
Key insights
- Synthetic data reliably predicts the *relative ranking* of different LLMs, suggesting its utility in selecting the best-performing model even when absolute scores differ from real data.
- The representativeness of synthetic data as a benchmark diminishes with increasing task complexity, being more effective for simpler tasks like intent detection than for complex ones such as Named Entity Recognition (NER).
- A quantifiable "bias factor" exists, indicating that LLMs tend to perform better on data they generated themselves, with smaller LLMs and more complex tasks showing slightly higher bias.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.