Reproduce “Light Alignment Improves LLM Safety via Model Self-Reflection with a Single Neuron” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sicheng Shen, Mingyang Lv, Han Shen, Jialin Wu, Binghao Wang, Zhou Yang, Guobin Shen, Dongcheng Zhao and 2 more
Researchers from the Beijing Institute of AI Safety and Governance introduce Neuron-Guided Safe Decoding (NGSD), an inference-time alignment method that improves large language model safety against jailbreak attacks while preserving utility. NGSD combines prompt-level self-reflection with a biologically inspired, single-neuron-based gating mechanism for adaptive, token-level safety intervention, achieving superior safety with reduced utility degradation and higher throughput.
Key insights
- Intelligently coupling prompt-level self-reflection with a token-level neuron-guided gating mechanism allows for adaptive and efficient safety interventions while preserving base model utility.
- A single, biologically inspired neuron can effectively integrate risk signals over time, activating safety interventions only for sustained deviations from safe behavior, which mitigates transient noise and reduces unnecessary interference.
- Training a lightweight safety expert model (the smallest in a family) on safety augmentation datasets enables efficient knowledge transfer and robust safety alignment to larger, related models with minimal cost.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.