Reproduce “ContextWeave: A Real-World Workflow Benchmark” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Bo Wang, Yuqian Yao, Enxi Wang, Luozhijie Jin, Yang Liu, Yiran Suo, Yuxuan Cai, Enyu Zhou and 20 more

Fudan University and collaborators introduced ContextWeave, a benchmark comprising 1,005 executable tasks derived from real-world, multi-month office workflows, designed to evaluate how memory systems impact agent performance in longitudinal, stateful contexts. Experiments show that memory consistently improves agent task completion quality and alignment with user preferences, shifting outcomes from basic completion to usable work.

Key insights

  • Memory systems that provide concrete, in-context experience (e.g., a-mem) are more effective than summary-oriented systems in improving agent performance, suggesting that actionable recall is crucial for workflow continuation.
  • The benefits and risks of memory are influenced by the actionability of recalled experience; richer recall can lead to higher task solvability but also increases susceptibility to misleading information.
  • While memory improves all tested base models, the extent of the gain varies, indicating that a model's inherent ability to translate recalled information into effective action is a factor in memory utility.
  • Models showing larger performance gains from memory tend to reallocate effort from exploration to execution, efficiently leveraging historical context rather than merely shortening trajectories.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.