Reproduce “Beyond Gaussian Bottlenecks: Topologically Aligned Encoding of Vision-Transformer Feature Spaces” on OpenResearch

About this paper

Andrew Bond, Ilkin Umut Melanlioglu, Erkut Erdem, Aykut Erdem

Researchers from Koç University and Hacettepe University introduced S²VAE, a variational autoencoder that compresses Vision Transformer (ViT) features by aligning the latent space with their intrinsic hyperspherical geometry using a product of spherical distributions. This approach achieved superior feature reconstruction and improved performance on geometric tasks like depth estimation, demonstrating an AbsRel of 0.04 for VGGT features compared to 0.07 for traditional Gaussian VAEs, particularly under high compression.

Key insights

• Vision Transformer features inherently possess a hyperspherical geometry, making angular relationships crucial for preserving semantic meaning during compression.
• Aligning the latent space topology with the data's intrinsic geometry (e.g., hyperspherical for ViT features) significantly improves feature reconstruction quality and downstream task performance, particularly under high compression.
• A product of multiple lower-dimensional spherical distributions effectively overcomes numerical instabilities of single high-dimensional spheres, allowing for robust and effective hyperspherical latent learning.

Summary by alphaXiv.