Reproduce “MetaOthello: A Controlled Study of Multiple World Models in Transformers” on OpenResearch

About this paper

Aviral Chawla, Galen Hall, Juniper Lovato

This research investigates how transformer models manage multiple "world models" when trained on heterogeneous data from diverse, potentially conflicting generative processes. It demonstrates that models learn shared, abstract representations of internal states, employing specialized internal circuits to dynamically select or suppress different rule systems based on the nature and context of conflicts.

Key insights

• Transformers learn a shared, 'Platonic' geometry of board-state representations that aligns across different game rules, including those with scrambled input tokenization (syntax invariance).
• Conflict resolution for ambiguous inputs is handled by specialized, localized neural circuits, rather than by entirely separate sub-models, demonstrating an economical use of capacity.
• The model employs different arbitration strategies depending on the nature of rule divergence: mid-layer reweighting for plausible, frequently ambiguous conflicts and early commitment/discarding for highly divergent or out-of-distribution scenarios.

Summary by alphaXiv.