Reproduce “Compositional Objectives: Learning Structure in Structure” on OpenResearch

About this paper

Pranavchandra Vivekananda, Sumukh Bettadapura, Ajan Subramanian

The work analyzes what spectral epiplexity objectives reward and finds that broad representation spectra do not guarantee task-relevant information. In ImageNet-1K experiments, a spectral objective using masked image views achieved the strongest results on most classification evaluations, while a separate compositional predictive objective made representations much more predictable to a fixed observer but generally did not improve classification.

Key insights

• A broad covariance spectrum does not identify which variables a representation encodes: representations with identical isotropic covariance can differ from perfect label prediction to chance performance.
• The masked-view spectral model had the broadest spectrum and led on most classification evaluations, whereas the compositional predictive objective produced the most concentrated spectrum but the strongest held-out observer predictability.
• Observer predictability, spectral spread, and downstream utility measure distinct properties; improving one does not necessarily improve the others.

Summary by alphaXiv.