Reproduce “Extracting and Understanding the Superficial Knowledge in Alignment” on OpenResearch

About this paper

Runjin Chen, Gabriel Jacob Perin, Xuxi Chen, Xilun Chen, Yan Han, Nina S. T. Hirata, Junyuan Hong, Bhavya Kailkhura

Researchers from The University of Texas at Austin, University of São Paulo, LinkedIn, and Lawrence Livermore National Laboratory developed a method to define, extract, and quantify "superficial knowledge" within LLM alignment. This work demonstrates that superficial knowledge largely eliminates safety and toxicity risks, contributes substantially to reasoning and factuality improvements, and is transferable across models, offering a robust solution for restoring compromised safety mechanisms.

Key insights

• A significant portion of LLM alignment, particularly for safety and detoxification, can be attributed to superficial knowledge that affects output style and format.
• Alignment is not entirely superficial; a performance gap remains for complex reasoning and factual understanding, indicating the necessity of deeper knowledge.
• Extracted superficial knowledge is modular, transferable across models (enabling weak-to-strong alignment), and effectively recoverable to restore safety in models compromised by adversarial attacks.

Summary by alphaXiv.