Reproduce “Iterative Finetuning is Mostly Idempotent” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zephaniah Roe, Jack Sanderson, Dang Nguyen, Julian Huang, Todd Nief, Aryan Shrivastava, Chenhao Tan, Ari Holtzman

Researchers from the University of Chicago investigated whether repeatedly finetuning large language models on their own outputs amplifies behavioral tendencies. They found that while supervised finetuning rarely led to amplification and often degraded coherence, direct preference optimization in a continual learning setup reliably amplified traits, though with subtle degradation in output structure.

Key insights

  • Iterative finetuning, especially in SFT and SDF settings, is largely idempotent or dissipative, indicating that traits typically persist or decay rather than amplifying significantly.
  • Amplification in SFT and SDF is rare, fragile, and almost always accompanied by a degradation in coherence, often resulting in repetitive or collapsed outputs.
  • Direct Preference Optimization (DPO), when models are continually updated from previous checkpoints, reliably induces trait amplification, where traits can progressively increase over cycles.
  • While continual DPO can amplify traits with relatively high superficial coherence scores, it still leads to subtle but measurable degradation in underlying response structure, such as decreased sentence length or diversity.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.