Reproduce “Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang and 12 more

The research introduces GLAN (Generalized LLM-Augmented iNstruction tuning), an approach for generating comprehensive instruction data for training Language Models by systematically decomposing a human knowledge taxonomy. This method reduces reliance on existing datasets or extensive human curation, requiring only a taxonomy as input. GLAN-tuned Mistral 7B models demonstrate improved performance in mathematical reasoning, coding, logical reasoning, and academic benchmarks, outperforming larger models like WizardLM-13B on certain tasks.

Key insights

  • Systematically decomposing a broad knowledge taxonomy can generate highly diverse and comprehensive instruction data "almost from scratch."
  • Leveraging LLMs in a multi-stage, hierarchical generation process can significantly reduce manual data curation compared to existing methods.
  • The generated education-inspired instruction data can improve an LLM's generalizability and performance across various domains.
  • The hierarchical approach ensures that the generated instructions are diverse, comprehensive, and educationally sound, covering a wider range of domains than previous methods relying on seed examples or existing datasets.
  • Ablation studies confirmed that the hierarchical decomposition of knowledge and the diversity of generated instructions were crucial to the performance gains, indicating the efficiency of the approach

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.