Reproduce “CompAct-Bench: A Benchmark for Working-Context Compaction in Long-Horizon Agent Tasks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiayi Chen, Lizhe Chen, Yujia Chen, Shihan Dou, Ming Zhang, Chendi Ge, Shiguang Ni, Tao Gui and 3 more

CompAct-Bench measures whether agents finish tasks after their interaction history is compacted into a shorter handoff. It evaluates 12 compactors on 200 tasks, with three continuations per context and a 10% character-length retention target. Mean post-compaction accuracy is 56.7%; Claude Opus 5 scores 68.5%. Tasks were pre-screened for successful un-compacted execution, so this measures compaction disruption, not general task-solving ability.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.