Reproduce “Alignment via Training Against Probes Without Losing Monitorability” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lena Libon, Alexander Panfilov, Ben Rank, Xin Chen, Jonas Geiping, Maksym Andriushchenko

The authors train language models against internal probes that are updated as activations change, using LoRA and a KL penalty to preserve utility. In tested harmfulness and honesty tasks, updated probes improved measured behavior, while fresh linear probes still detected target concepts. Frozen probes failed, and benign-only further fine-tuning undid safety gains in one tested case.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.