Reproduce “Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicity” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rheeya Uppaal, Apratim Dey, Yiting He, Yiqiao Zhong, Junjie Hu

ProFS (Projection Filter for Subspaces) presents a tuning-free LLM alignment method that reduces toxicity by directly modifying MLP-value layer weights through factor analysis and SVD. This approach demonstrates superior sample efficiency and robustness to noisy preference data compared to DPO, while effectively preserving the model's general capabilities.

Key insights

  • Direct model editing through subspace projection offers a highly sample-efficient and robust alternative to gradient-based fine-tuning for LLM safety alignment.
  • The method's effectiveness hinges on a centering technique that isolates undesirable behavioral components from essential linguistic knowledge encoded in embeddings.
  • ProFS theoretically connects model editing to DPO, showing it can act as a denoised version of a single DPO optimization step, providing new theoretical links between alignment paradigms.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.