Reproduce “Chunky Post-Training: Data Driven Failures of Generalization” on OpenResearch

About this paper

Seoirse Murray, Allison Qi, Timothy Qian, John Schulman, Collin Burns, Sara Price

The authors introduce SURF to find misrouted model behaviors and TURF to link failures with training-data patterns. TURF links Tülu 3 failures to supervised fine-tuning data, and data ablations alter a rebuttal failure. On 1,500 BIG-Bench reasoning questions, a surface-format change lowers Tülu 3 accuracy from 57.7% to 49.8%, but Qwen2-7B changes little; this does not identify the cause.

Key insights

• Many seemingly emergent, unintended LLM behaviors are actually data-driven, stemming from models learning spurious correlations from discrete post-training datasets.
• The underspecification in behavioral routing, where training data specifies *what* behavior should occur but not its boundaries, often leads to overgeneralization based on superficial cues.
• Automated tools like SURF and TURF offer a systematic approach to both discover these generalization failures and trace them back to specific data patterns, enabling root cause analysis.

Summary by alphaXiv.