Reproduce “The Model Agreed, But Didn't Learn: Diagnosing Surface Compliance in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaojie Gu, Ziying Huang, Weicong Hong, Jian Xie, Renze Lou, Kai Zhang
A diagnostic framework, Self-Assessment Multiple Choice Question (SA-MCQ), uncovers that large language models frequently demonstrate surface compliance to knowledge edits, where models mimic updated behavior without genuinely altering internal memory. This work further reveals the fragility of these superficial edits when exposed to varying external evidence and recursive updates.
Key insights
- A significant disconnect, termed "Surface Compliance," exists where high performance on traditional generative metrics often does not correlate with genuine internal belief modification, as revealed by discriminative assessments like SA-MCQ.
- Edited models demonstrate heightened sensitivity to external contextual evidence, often disrupting the inertia of original parametric weights rather than precisely overwriting knowledge, making them susceptible to various informational cues.
- Recursive knowledge modifications lead to "persistent representational residues," causing a progressive degradation in memory stability and reversibility, making subsequent updates less effective and leaving the model in a "metastable state" with cognitive instability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.