Reproduce “Learning Sparse Visual Representations via Spatial-Semantic Factorization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Theodore Zhengde Zhao, Sid Kiblawi, Jianwei Yang, Naoto Usuyama, Reuben Tan, Noel C Codella, Tristan Naumann, Hoifung Poon and 1 more

STELLAR introduces Spatial-Semantic Factorization to learn sparse visual representations, resolving the "Invariance Paradox" by disentangling semantic concepts from spatial locations. The method achieves a unified representation that simultaneously excels at high-level semantic understanding (e.g., 79.10% ImageNet linear accuracy for ViT-H) and low-level pixel reconstruction (e.g., 2.60 FID for ViT-H), reducing latent representation size by up to 90%.

Key insights

  • The Invariance Paradox can be resolved by explicitly disentangling semantic (invariant) and spatial (equivariant) information within a factorized latent representation.
  • A sparse, low-rank representation (e.g., 16 tokens for ViT-B) can maintain a strong balance between reconstruction fidelity and semantic richness, despite significantly reducing latent dimensionality.
  • Multi-objective training with reconstruction, clustering, and optimal transport-based alignment is crucial for learning robust, view-invariant semantic concepts and holistic sparse tokens.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.