Reproduce “Probing Persona-Dependent Preferences in Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Oscar Gilg, Pierre Beckmann, Daniel Paleka, Patrick Butlin

The authors train a linear probe on internal activations to predict task utilities fitted from pairwise choices. The probe predicts preferences across personas, and its harmful-versus-benign score reverses under an evil persona. On Gemma-3-27B, it predicts evil utilities better than Assistant utilities do, and steering changes task choices. Findings cover Gemma-3-27B and Qwen-3.5-122B; steering evidence is limited to Gemma.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.