Reproduce “On the Sentence Embeddings from Pre-trained Language Models” on OpenResearch

About this paper

Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, Lei Li

A study by researchers at ByteDance AI Lab and Carnegie Mellon University empirically diagnoses the anisotropic nature of BERT's un-fine-tuned sentence embedding space and introduces BERT-flow, an unsupervised method using normalizing flows to transform these embeddings into a standard Gaussian distribution. This calibration consistently improves performance on semantic textual similarity tasks, yielding up to an 8.16 average gain in Spearman's correlation on STS benchmarks and establishing new state-of-the-art results when integrated with NLI fine-tuning.

Key insights

• BERT's pre-training objective implicitly encourages semantically meaningful representations, but these are not effectively exploited by standard similarity metrics due to the anisotropic and biased structure of the embedding space.
• The raw BERT sentence embedding space is biased by word frequency, with high-frequency words clustering near the origin, and exhibits sparsity for low-frequency words, creating 'holes' that degrade semantic similarity performance.
• Transforming the BERT embedding distribution into a smooth, isotropic standard Gaussian space via unsupervised normalizing flows effectively mitigates these geometric issues, making inherent semantic information accessible without discarding it.

Summary by alphaXiv.