Reproduce “SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models” on OpenResearch

About this paper

Bin Wang, C.-C. Jay Kuo

The SBERT-WK method introduces a training-free approach to generate universal sentence embeddings by geometrically analyzing how word representations evolve across the layers of pre-trained BERT-based models. This approach dynamically weights contributions from different layers and word importance, yielding improved performance on semantic textual similarity, downstream classification, and linguistic probing tasks.

Key insights

• Word representations in BERT-based models exhibit distinct evolution patterns across layers, with intermediate layers showing stability and higher layers showing rapid changes, suggesting a spectrum of linguistic information.
• The variability (variance) of a word's representation across layers correlates with its semantic informativeness; words with richer content show greater variance.
• By fusing layer-wise information using geometric properties (alignment and novelty) and incorporating word importance derived from this variability, higher quality universal sentence embeddings can be extracted without additional training.

Summary by alphaXiv.