Reproduce “Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning” on OpenResearch

About this paper

Zachary Shinnick, Liangze Jiang, Hemanth Saratchandran, Anton van den Hengel, Damien Teney

This research explores how transformers acquire algorithmic reasoning abilities when pretrained on simple, synthetic procedural data. The study identifies that distinct procedural tasks instill specialized capabilities, localizes these learned structures within specific architectural components (attention or MLP layers), and demonstrates their precise, modular, and composable nature, paving the way for disentangling knowledge from reasoning in AI models.

Key insights

• Different forms of procedural pretraining data instill distinct and specialized algorithmic reasoning capabilities, acting as 'soft inductive biases' tailored to the data's structure.
• Beneficial inductive biases are often localized within specific architectural components, with attention layers frequently carrying the most transferable information for generalizable algorithmic reasoning patterns.
• The improvements from procedural pretraining are due to precise structural encoding in the weights, as evidenced by significant performance degradation from noise and shuffling, indicating non-trivial learning.
• Inductive structures learned from distinct procedural tasks are modular and can be effectively combined, suggesting a path towards composing specialized reasoning modules within a single model.

Summary by alphaXiv.