Reproduce “Accuracy is Not All You Need” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, Ramachandran Ramjee
Microsoft Research Bangalore highlights that evaluating compressed large language models solely on accuracy overlooks "flips," where model answers change from the baseline while overall accuracy is maintained. The research introduces %Flips and KL-Divergence as more sensitive metrics, demonstrating that high flip rates correlate with degraded performance on generative tasks.
Key insights
- Accuracy metrics alone are insufficient to evaluate compressed LLMs, as they can mask significant behavioral changes.
- The percentage of "flips" (answer changes) increases much faster than accuracy degradation under aggressive compression.
- Questions with low "Top Margin" (small confidence gap between top choices) are most prone to flipping and are often incorrect in the baseline.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.