Reproduce “Portable Reward Tuning: Towards Reusable Fine-Tuning across Different Pretrained Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Daiki Chijiwa, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito, Susumu Takeuchi
NTT Corporation introduces Portable Reward Tuning (PRT), a method that enables the reuse of fine-tuning efforts across different pretrained foundation models by reformulating fine-tuning as a reward maximization problem. PRT achieves comparable or superior performance to traditional fine-tuning and Emulated Fine-Tuning while significantly reducing inference overhead by requiring only a single additional reward model at inference.
Key insights
- Fine-tuning can be re-conceptualized as learning a portable reward function, rather than directly modifying the parameters of a foundation model.
- KL-divergence regularization with respect to the pretrained model allows for an analytical derivation of the PRT model, enabling single-model inference.
- PAC-Bayesian generalization bounds and optional entropy maximization provide theoretical grounding and enhance PRT's generalization capability across diverse pretrained models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.