Reproduce “CompAct-Bench: A Benchmark for Working-Context Compaction in Long-Horizon Agent Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiayi Chen, Lizhe Chen, Yujia Chen, Shihan Dou, Ming Zhang, Chendi Ge, Shiguang Ni, Tao Gui and 3 more
CompAct-Bench measures whether agents finish tasks after their interaction history is compacted into a shorter handoff. It evaluates 12 compactors on 200 tasks, with three continuations per context and a 10% character-length retention target. Mean post-compaction accuracy is 56.7%; Claude Opus 5 scores 68.5%. Tasks were pre-screened for successful un-compacted execution, so this measures compaction disruption, not general task-solving ability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.