Reproduce “A Distributional Optimisation Perspective on Combining Models in Deep Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Congye Wang, Yan Lin, Zheyang Shen, Matthew A. Fisher, Chris. J. Oates
The authors optimize model collections as distributions over parameters, training candidate models against their combined prediction rather than combining independently trained models afterward. For Llama-3-8B rank-256 adapters across eight commonsense tasks, VGD achieved an equally weighted mean accuracy of 83.54%, versus 81.83% for MFLD and 82.75% for standard fine-tuning. The proof-of-concept result is limited to this tested setup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.