Reproduce “Embedding-to-Prefix: Parameter-Efficient Personalization for Pre-Trained Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bernd Huber, Ghazal Fazelnia, Andreas Damianou, Sebastian Peleato, Max Lefarov, Praveen Ravichandran, Marco De Nadai, Mounia Lalmas-Roellke and 1 more
Embedding-to-Prefix (E2P), developed by Spotify, is a parameter-efficient method for personalizing large language model outputs by projecting dense user embeddings into a single soft token prefix. This approach significantly improves the relevance and quality of generated content across dialogue, news headlines, and large-scale music and podcast recommendations while keeping the LLM backbone frozen.
Key insights
- A single soft prefix token derived from a meaningful user embedding can significantly improve the relevance and quality of LLM-generated content with minimal computational overhead.
- E2P effectively leverages existing, high-quality user representation learning infrastructure by directly integrating dense user embeddings into LLMs.
- Dense embedding-based personalization through E2P provides complementary signals that can enhance and be combined with other contextual inputs, such as textual prompts, for richer user experiences.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.