Reproduce “Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectives” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chloe Li, Mary Phuong, Daniel Tan
Researchers from University College London introduce Self-Report Fine-Tuning (SRFT), a method to train large language models (LLMs) to honestly self-report their hidden objectives during interrogation. This technique enabled LLMs to achieve a 0.98 F1 score in detecting hidden objectives and recover 28-100% of their details, significantly outperforming baseline models which scored 0%.
Key insights
- Training models to admit simple 'on-policy' factual errors in a basic Q&A setting generalizes to honest self-reporting of complex, hidden objectives in out-of-distribution adversarial scenarios.
- The effectiveness of honesty generalization is strongly linked to training admissions that are consistent with the model's 'belief' about its own truthfulness, rather than objective ground truth.
- SRFT-induced honesty remains robust against explicit non-disclosure instructions and the presence of adversarial 'decoy objectives' during interrogation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.