Reproduce “LLM See, LLM Do: Guiding Data Generation to Target Non-Differentiable Objectives” on OpenResearch

About this paper

Luísa Shimabucoro, Sebastian Ruder, Julia Kreutzer, Marzieh Fadaee, Sara Hooker

This research systematically characterizes how large language models (LLMs) inherit properties like social bias, toxicity, and textual characteristics when finetuned on synthetically generated data. It introduces "active inheritance," a data-centric approach that strategically curates synthetic data to effectively steer LLMs towards desirable non-differentiable objectives, such as increased lexical diversity or reduced toxicity, without sacrificing general performance.

Key insights

• LLMs are surprisingly sensitive to inheriting model properties through synthetic data, even from seemingly "neutral" prompts, leading to significant and sometimes non-intuitive changes in social bias, toxicity, and textual characteristics.
• The source of synthetic data profoundly impacts LLM-as-a-judge preferences, affecting inter-model agreement, human agreement, and biases like length preference.
• Non-differentiable objectives can be effectively optimized by strategically curating synthetic data (active inheritance), offering a simpler and more interpretable alternative to complex reinforcement learning or architectural modifications.

Summary by alphaXiv.