Reproduce “ULLME: A Unified Framework for Large Language Model Embeddings with Generation-Augmented Learning” on OpenResearch

About this paper

Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Thien Huu Nguyen

ULLME introduces a unified framework for generating large language model embeddings that integrates diverse LLM architectures and fine-tuning strategies. Its novel Generation-augmented Representation Learning (GRL) method consistently achieves state-of-the-art performance on the MTEB benchmark across various LLMs, improving average scores by up to 2.03 points over prior methods like LLM2Vec.

Key insights

• Enabling bidirectional attention in causal large language models is crucial for their effective application in dense retrieval tasks, allowing full contextual understanding.
• Leveraging large language models' inherent generative capabilities through Generation-augmented Representation Learning (GRL) can significantly enhance their discriminative embedding performance by explicitly aligning embedding-based and generation-based relevance scores.
• A unified framework that supports diverse LLM architectures and integrates multiple fine-tuning strategies accelerates research and practical application by simplifying complex model adaptation and evaluation.

Summary by alphaXiv.