Reproduce “MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jeremy Carleton, Debajoy Mukherjee, Srinivas Shakkottai, Dileep Kalathil
MAVIS steers a frozen language model toward weighted objectives without changing generator weights. It trains separate value models for each objective and uses estimates to reweight next tokens, allowing users to adjust objective weights at inference time. On Anthropic HH-RLHF, Llama-2 7B scored 2.311 for helpfulness versus PPO’s 2.104, at higher approximate KL; these are reward-model scores, not human ratings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.