Reproduce “LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lanxin Zhao, Bamdev Mishra, Pratik Jawanpuria, Lequan Lin, Dai Shi, Junbin Gao, Andi Han
LOFT adapts pretrained linear layers by choosing task-informed input directions, then learning an orthogonal transform within them. It estimates supports from downstream gradients. In a matched six-task DeBERTaV3-base GLUE comparison, rank-46 SkewGrad scored 88.73 versus 88.06 for principal-support LOFT, with equal trainable parameters and training-loop memory. The averages are descriptive, and fixed support storage is excluded from those parameter counts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.