Reproduce “Offline bilingual word vectors, orthogonal transformations and the inverted softmax” on OpenResearch

About this paper

Samuel L. Smith, David H. P. Turban, Steven Hamblin, Nils Y. Hammerla

This work formalizes and refines the 'offline' approach to bilingual word vectors by proving that the optimal linear mapping between word vector spaces is an orthogonal transformation, solvable via Singular Value Decomposition (SVD). It also introduces an 'inverted softmax' to mitigate the 'hubness problem' in high-dimensional nearest-neighbor retrieval, achieving up to 43.1% precision@1 for English-Italian word translation and demonstrating robustness when trained with non-expert pseudo-dictionaries and extending to sentence retrieval with 67.8% precision@1.

Key insights

• A self-consistent linear transformation between two semantic word vector spaces is proven to be orthogonal, and Singular Value Decomposition (SVD) provides an exact solution for learning this optimal mapping.
• The 'hubness problem' in high-dimensional nearest-neighbor retrieval can be effectively mitigated by the proposed 'inverted softmax' method, leading to improved translation precision.
• Robust bilingual word vector alignment is achievable using easily obtainable, non-expert 'pseudo-dictionaries' derived from identical character strings or aligned sentences, reducing reliance on expensive expert resources.

Summary by alphaXiv.