Reproduce “Reasoning over Boundaries: Enhancing Specification Alignment via Test-time Deliberation” on OpenResearch

About this paper

Haoran Zhang, Yafu Li, Xuyang Hu, Dongrui Liu, Zhilin Wang, Bo Li, Yu Cheng

This paper defines "specification alignment" to address how large language models (LLMs) adhere to dynamic, scenario-specific behavioral and safety guidelines. The SPECBENCH benchmark was developed to evaluate this, and ALIGN3, a three-stage test-time deliberation method, enhanced alignment rates for models like Qwen3-14B by 11.89% with low computational cost.

Key insights

• The inherent "safety-behavior trade-off" was consistently observed, where models prioritizing safety often over-refused, reducing helpfulness, and vice-versa.
• Reasoning-capable LLMs generally exhibit better specification alignment than instruct-tuned models, indicating that explicit deliberation processes aid adherence to complex guidelines.
• The difficulty of achieving specification alignment varies across scenarios, with Code and Biochemical Procedure Instruction being particularly challenging due to stricter safety requirements and ambiguous intentions.

Summary by alphaXiv.