Reproduce “Hyper-Compression: Model Compression via Hyperfunction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fenglei Fan, Juntong Fan, Dayang Wang, Jingbo Zhang, Zelin Dong, Shijun Zhang, Ge Wang, Tieyong Zeng
Hyper-Compression tackles the storage demands of large models by encoding pairs of weights as integer indices into a finite trajectory-based codebook, then reconstructing approximate values without calibration data. On Qwen1.5-7B-Chat, it uses 5.79 GB and scores 64.35 across eight tasks, versus GPTQ-INT4 at 5.86 GB and 64.12. The result is a trade-off in this evaluation, not a general guarantee.
Key insights
- Model compression can be formulated as deterministic coordinate coding: a high-dimensional collection of weights is represented by low-dimensional trajectory parameters plus integer locations.
- The ergodic representation provides dense coverage only asymptotically, so finite-codebook quality depends on trajectory uniformity, search range, scaling, and the distribution of the original weights.
- Hyper-Compression is complementary to structural compression: combining it with pruning raises total ratios substantially, but the additional approximation error can become pronounced for already-small or heavily compressed models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.