Reproduce “Benchmarking and Rethinking Knowledge Editing for Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Guoxiu He, Xin Song, Futing Wang, Aixin Sun

A comprehensive benchmarking study on knowledge editing for large language models found that most parameter-modification methods perform poorly under realistic autoregressive inference and sequential editing, often degrading reasoning abilities. In contrast, a simple external memory-based approach, Selective Contextual Reasoning (SCR), demonstrated superior stability and effectiveness across various LLMs and knowledge types.

Key insights

  • Most parameter-modification knowledge editing methods perform poorly under realistic autoregressive inference and sequential editing, often failing to retain knowledge and degrading LLM reasoning abilities.
  • Non-parametric approaches, such as in-context learning and external memory (e.g., SCR), demonstrate superior stability and effectiveness, particularly for complex event-level knowledge and reasoning-oriented LLMs.
  • A fundamental trade-off exists between the speed of knowledge integration (parameter-editing methods) and the minimal additional inference latency (context-based methods), with no single method excelling in both.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.