Reproduce “A Quantitative Characterization of Forgetting in Post-Training” on OpenResearch

About this paper

Krishnakumar Balasubramanian, Shiva Prasad Kasiviswanathan

Researchers from UC Davis and Amazon quantitatively characterized forgetting in generative model post-training pipelines, establishing that Forward-KL objectives inherently cause mass forgetting while Reverse-KL objectives naturally preserve old knowledge with controlled component drift. The study reveals distinct roles for replay mechanisms based on the chosen divergence objective.

Key insights

• Forward-KL objectives inherently lead to mass forgetting (complete discarding of old modes) when trained solely on new data, even if the model has the capacity to retain them.
• Reverse-KL objectives are naturally aligned with knowledge retention, as their population minimizers prevent mass forgetting and maintain the correct proportions of old knowledge.
• Old-component drift in Reverse-KL objectives is gated by the geometric separation of modes, becoming exponentially small when modes are sufficiently distinct.
• Replay plays fundamentally different roles: for Forward-KL, it modifies the population objective itself, while for Reverse-KL, it primarily stabilizes stochastic optimization by ensuring old-mode visibility in minibatches.

Summary by alphaXiv.