Reproduce “Whitening Not Recommended for Classification Tasks in LLMs” on OpenResearch

About this paper

Ali Forooghi, Shaghayegh Sadeghi, Jianguo Lu

Researchers at the University of Windsor found that applying whitening transformations to Large Language Model embeddings consistently reduced performance on classification tasks and exhibited inconsistent benefits for semantic text similarity across various LLMs. The study also introduced SentEval+, a platform facilitating efficient LLM embedding evaluation by providing pre-computed embeddings.

Key insights

• Whitening transformations, while successfully improving embedding isotropy, consistently and universally degrade the performance of embeddings on classification tasks across all tested models and datasets.
• The benefits of whitening for Semantic Text Similarity (STS) tasks are model-dependent, showing improvements for some base models but no benefits or even deterioration for fine-tuned models like AnglE-BERT/LLaMA, SimCSE, and notably ChatGPT embeddings.
• A fundamental disconnect exists between improved embedding isotropy (as measured by IsoScore) and actual performance on downstream machine learning tasks, indicating that greater isotropy does not always equate to increased utility for all applications.

Summary by alphaXiv.