Reproduce “DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ying Zhou, Xinyao Wang, Yulei Niu, Yaojie Shen, Lexin Tang, Fan Chen, Ben He, Le Sun and 1 more
DiffLM, a framework from the University of Chinese Academy of Sciences and ByteDance, combines VAEs, diffusion models, and frozen large language models to generate high-quality synthetic structured data. It produces data for tabular, code, and tool generation tasks that often outperforms existing LLM-based methods and, in some cases, real data for downstream model performance, demonstrating improved diversity and fidelity.
Key insights
- Decoupling the learning of complex data distributions via VAEs and diffusion models from the LLM's generative objective enhances the LLM's ability to produce diverse and faithful synthetic data.
- An adaptive decreasing beta annealing strategy in VAE training is crucial for developing a robust and effective latent space that accurately represents the data distribution.
- Soft prompt latent feature injection provides an efficient and non-intrusive mechanism to guide frozen LLMs, preserving their general knowledge while enabling controllable structured data generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.