Reproduce “Object Scene Representation Transformer” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mehdi S. M. Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetić, Mario Lučić, Leonidas J. Guibas, Klaus Greff and 1 more

An Object Scene Representation Transformer (OSRT) from Google Research enables unsupervised 3D-consistent object decomposition and novel view synthesis, achieving over 3000x faster rendering and significantly higher accuracy on complex scenes compared to prior methods.

Key insights

  • The Slot Mixer decoder is a critical architectural innovation, enabling orders of magnitude faster compositional rendering (over 30x speedup over Spatial Broadcast decoders) by decoupling slot processing from per-pixel decoding.
  • Novel View Synthesis serves as a crucial unsupervised auxiliary task, compelling the model to learn geometrically consistent 3D object structures necessary for effective scene decomposition.
  • OSRT demonstrates that combining efficient light-field-based scene representation with object-centric inductive biases can scale unsupervised 3D object decomposition to highly complex and photo-realistic environments.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.