Reproduce “LLM Safety From Within: Detecting Harmful Content with Internal Representations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Difan Jiao, Yilun Liu, Ye Yuan, Zhenwei Tang, Linfeng Du, Haolun Wu, Ashton Anderson
A framework named SIREN detects harmful content in large language models by systematically utilizing their internal representations. This approach achieves superior performance, generalization, and efficiency compared to existing generative guard models.
Key insights
- Safety-relevant information is not solely present in an LLM's final output layer but is distributed and sparsely encoded across its internal representations.
- Systematically combining features from adaptively weighted safety neurons across multiple internal layers yields a more robust and consistent understanding of harmfulness.
- Leveraging internal representations through a lightweight, plug-and-play framework offers substantial improvements in detection efficacy, generalization, and computational efficiency over fine-tuned generative guard models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.