Reproduce “Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning” on OpenResearch

About this paper

Shuai Tang, Virginia R. de Sa

An invertible decoder turns the prediction component of an unsupervised sentence encoder into a second usable representation rather than discarding it after training. Combining encoder and inverse-decoder features improves sentence similarity performance over FastText averaging, reaching average Pearson correlations of up to 71.1, while showing that the orthonormal constraint is important for effective representation fusion.

Key insights

• The decoder can remain useful after training: its inverse produces sentence representations that are competitive with, and sometimes stronger than, those from the recurrent encoder.
• The encoder and inverse decoder capture complementary information; combining them raises the BookCorpus linear model’s STS average from 64.9 for the encoder alone and 67.6 for the inverse decoder alone to 70.2.
• Invertibility matters more than decoder complexity: linear and nonlinear bijective decoders perform similarly, while removing the orthonormal constraint substantially weakens the encoder and the combined representation.

Summary by alphaXiv.