Reproduce “CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Morayo Danielle Adeyemi, Ryan A. Rossi, Franck Dernoncourt
CAVEWOMAN compares compressing a prompt with constraining a model’s response on the same models and items, measuring accuracy, agreement with unconstrained text, and realized per-item API cost. At the telegraphic output level, 17 of 20 API model-by-dataset cells are cheaper. Input compression usually raises net cost, but GPT-4o is an exception, and hidden reasoning tokens limit this result for GPT-5.4.
Key insights
- Output compression consistently reduces realized per-item costs for LLMs, while input compression often leads to an increase in net costs due to compensatory output expansion.
- LLMs can achieve correct answers under output compression while their generated surface text significantly diverges from their unconstrained outputs, indicating a decoupling of accuracy from semantic fidelity.
- Model robustness to linguistic compression varies widely across different LLMs and is not directly correlated with their parameter count or unconstrained baseline performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.