Reproduce “LLMs Corrupt Your Documents When You Delegate” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Philippe Laban, Tobias Schnabel, Jennifer Neville

A new benchmark, DELEGATE-52, combined with a novel simulation methodology, evaluates Large Language Models (LLMs) for long-horizon, multi-session delegated document editing across 52 professional domains. Experiments with 19 LLMs revealed all models experience significant document corruption, averaging 50% loss over 20 interactions, with critical failures accounting for most degradation and current agentic tools failing to improve performance.

Key insights

  • LLM degradation in long-horizon delegated tasks is dominated by sparse but severe critical failures (10%+ score drop in one round-trip), rather than numerous small errors.
  • Short-term LLM performance after a single round-trip is not reliably predictive of long-horizon reliability in extended delegated workflows.
  • Implementing a basic agentic harness with file system and Python tools did not improve LLM performance for complex editing tasks, often leading to increased degradation and overhead.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.