Reproduce “When Bigger Becomes Brittle: Why Compression Hurts More as Language Models Scale” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gautam Gupta, Dikshant Kukreja, Unique Kumar, Avni Mittal, Deepanshu Dabas, Kshitij Sah, Erik Cambria, Rajiv Ratn Shah and 2 more
The authors study how targeted retrieval-head damage affects in-context retrieval on templated IOI prompts. They separate lost direct contribution from the network’s response, finding that larger Pythia models leave more damage visible at the output. Under simulated per-channel 4-bit quantization, H/D rises overall from 0.55 for Pythia-410M to 0.91 for Pythia-12B, with intermediate variation; this does not establish deployment robustness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.