Reproduce “TreeSynth: Synthesizing Diverse Data from Scratch via Tree-Guided Subspace Partitioning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sheng Wang, Pengan Chen, Jingqi Zhou, Qintong Li, Jingwei Dong, Jiahui Gao, Boyang Xue, Jiyue Jiang and 2 more
TREESYNTH presents a framework that synthesizes diverse, high-quality datasets for large language models from scratch using a tree-guided subspace partitioning approach. This method generates data with enhanced diversity, resulting in over 10% average performance improvement for fine-tuned models on various benchmarks, and demonstrates robust scalability.
Key insights
- Adopting a global, tree-guided subspace partitioning strategy can effectively overcome issues of data repetition and space collapse inherent in local, incremental data synthesis methods.
- Leveraging LLMs to determine core partitioning criteria and expand attribute values allows for systematic and comprehensive exploration of the entire data domain.
- Structured synthesis within explicitly defined atomic subspaces ensures fine-grained control over data diversity and distribution, leading to more balanced and robust datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.