Reproduce “GACC-Bench: Benchmarking Goal-Adaptive Context Compression Ability across Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Weiting Liu, Zhiyu Pan, Shuhong Wang, Zhenquan Yao, Jungang Li, Li Zhao, Jian Jiang, Jianfeng Feng and 2 more
The authors introduce GACC-Bench, which evaluates whether compressors can use a high-level goal and token budget to retain context for multiple withheld questions. It covers 500 goals and 2,571 questions across 281 contexts. At a 5% budget, target-goal compression reaches 27.04% Micro accuracy, versus 12.29% without a goal. Yet full-benchmark Macro accuracy peaks at 14.80%, showing complete goal-level coverage is limited.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.