Reproduce “Pref-CTRL: Preference Driven LLM Alignment using Representation Editing” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Imranul Ashrafi, Inigo Jauregi Unanue, Massimo Piccardi

Pref-CTRL introduces a framework for aligning Large Language Model outputs by enhancing representation editing with explicit human preference signals during inference. This method leverages a multi-objective value function training to achieve improved win rates and stronger generalization capabilities over baseline test-time alignment methods, without incurring the computational costs of full model fine-tuning.

Key insights

  • Integrating explicit human preference signals directly into the value function's training, via a Margin Loss, significantly improves the effectiveness and robustness of test-time LLM alignment.
  • The inclusion of a Regularization Loss is crucial for preventing the preference-driven value function from over-optimizing or 'reward hacking,' thereby ensuring the generated text remains fluent and coherent.
  • Combining a preference-aware value function with representation editing provides a computationally efficient and scalable method for LLM alignment that achieves performance competitive with resource-intensive fine-tuning approaches.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.