Reproduce “Quantization Effects on Tool-Failure Recovery Vary Across Prompts and Evaluation Designs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuhe Hu
The authors compare 8-bit and 4-bit variants of two language models on 20 synthetic order-management tasks and five prompts, testing recovery after a tool call fails once. There is no stable winner: Llama/P4 full-pipeline results favor 8-bit by 28.3 percentage points with tolerant scoring, but favor 4-bit by 15 points under strict rescoring; findings are limited to these settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.