Reproduce “Compressing and Interpreting Word Embeddings with Latent Space Regularization and Interactive Semantics Probing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haoyu Li, Junpeng Wang, Yan Zheng, Liang Wang, Wei Zhang, Han-Wei Shen
Ohio State University and Visa Research introduced a method that employs beta-Variational Autoencoders to compress word embeddings into a more compact and semantically organized latent space by identifying and removing redundant dimensions. An accompanying interactive visual analytics system reveals how specific semantics are encoded in these reduced dimensions, demonstrating competitive performance on NLP tasks while achieving significant dimensionality reduction.
Key insights
- betaVAE's regularization term effectively drives certain latent dimensions to become 'deprecated,' converging to unit Gaussians and losing semantic information, which allows for substantial compression without significant performance loss.
- The useful, non-deprecated dimensions in betaVAE concentrate semantics into fewer, more salient dimensions compared to standard Autoencoders, leading to a more structured and interpretable latent space.
- A perturbation-based approach, measuring angular deviation from user-defined semantic directions, effectively quantifies and visualizes the encoding-level of specific semantics within individual latent dimensions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.