Reproduce “DeepEdit: Knowledge Editing as Decoding with Constraints” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yiwei Wang, Muhao Chen, Nanyun Peng, Kai-Wei Chang

DeepEdit introduces a knowledge editing method that directly guides multi-step reasoning in large language models (LLMs) through constrained decoding. It substantially improves question answering accuracy when incorporating new knowledge, achieving up to a 69% gain on GPT-3.5-TURBO-INSTRUCT, while also reducing reasoning steps and generation time using an efficient depth-first search approach.

Key insights

  • Direct output control through carefully designed decoding constraints is an effective paradigm for knowledge editing in LLMs, significantly enhancing the logical coherence and precise integration of new information during multi-step reasoning.
  • Each of the four proposed constraints (Conciseness, Coherence, Receptiveness, Pertinence) contributes uniquely and critically to improving different aspects of the LLM's reasoning process, indicating that LLMs do not inherently satisfy these properties without explicit guidance.
  • An efficient depth-first search strategy, combined with an importance estimator and early-stopping, can drastically reduce the computational cost of constrained decoding for multi-hop reasoning while maintaining high accuracy, making the approach practical for black-box LLMs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.