Reproduce “Combining Domain and Alignment Vectors to Achieve Better Knowledge-Safety Trade-offs in LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Megh Thakkar, Quentin Fournier, Matthew Riemer, Pin-Yu Chen, Amal Zouaq, Payel Das, Sarath Chandar

MergeAlign combines domain and alignment weight changes from models sharing a pretrained base, producing a specialist checkpoint without another preference-training run. In medicine tests, it scores 99.67 on BeaverTails safety and 61.33 across five medical benchmarks, versus 67.8 and 61.37 for the domain expert. These benchmark results depend on a strong aligned sibling and do not guarantee safety in deployment.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.