Reproduce “Generalization properties of contrastive world models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kandan Ramakrishnan, R. James Cotton, Xaq Pitkow, Andreas S. Tolias
The authors test whether CSWM preserves object-wise representations under attribute shifts in synthetic environments. CSWM encodes observations into object slots and predicts changes after actions. On 2D shapes, prediction worsens with more out-of-distribution objects and longer rollouts, alongside qualitative evidence of object mixing. This supports, but does not prove, a representation-based explanation; other models were not tested.
Key insights
- Current contrastive world models, like CSWM, do not inherently generalize well to out-of-distribution data, challenging the assumption that self-supervised object-centric learning directly leads to robust generalization.
- The primary cause for OOD performance deterioration is a fundamental breakdown in the object encoder's ability to maintain factorized (disentangled) representations when encountering novel object attributes or their combinations.
- Failure in representation factorization directly leads to inaccurate state transitions, where the transition model updates states for multiple objects even when only one object is acted upon.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.