Reproduce “UTA: Universal Training Adaptive Learning Rate Decay” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anda Tang, Junjie Wang, Xuanlin Yang, Cong Fang, Zhouchen Lin
UTA adapts learning-rate decay to optimizer state, using accumulated learning-rate effort, called intrinsic time, to set schedule scale. Its schedule is derived through segment-wise, continuous-time analysis. For a 130M-parameter LLaMA trained for 15,000 steps, UTA reaches validation perplexity 20.16, versus 20.83 for WSD and 21.12 for UBA, showing lower perplexity at this reported budget, not lower compute.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.