Reproduce “Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges” on OpenResearch

About this paper

Nayoung Lee, Ziyang Cai, Avi Schwarzschild, Kangwook Lee, Dimitris Papailiopoulos

The authors present a self-improvement method that extends models’ performance to harder tasks. The method starts with labeled easy examples, then trains on model-generated answers to progressively harder controlled inputs, with filtering. On reverse addition, a transformer trained on examples from 1 to 16 digits generalizes near-perfectly to 100-digit problems. In multiplication, combined filtering reaches 100% accuracy on 6-by-6, in this comparison.

Key insights

• Unsupervised data filtering, such as length filtering and majority voting, is essential for successful self-improvement on complex tasks like forward addition and multiplication, preventing performance degradation.
• Standard transformer architectures, even without positional embeddings, can achieve extreme length and easy-to-hard generalization through iterative self-improvement, challenging the necessity of architectural modifications.
• The self-improvement process benefits significantly from pretrained models and can be accelerated by allowing the model to sample problems from multiple difficulty levels per round.

Summary by alphaXiv.