Reproduce “Benchmarks Are Surprisingly Compaction-Friendly” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jeanne Salle, Hardik Bhatnagar, Matthias Bethge, Sahar Abdelnabi, Maksym Andriushchenko
The authors test context compaction’s effects on performance and instruction retention in long-running agents, using OpenCode and five strategies. Scores change little across thresholds, while summaries omit an average of 0.92 of three anti-cheating rule categories after one compaction and 1.93 after ten. In 44 sessions, process and scope instructions were dropped more often than feature requirements; outcomes were not measured.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.