Reproduce “Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Omatharv Bharat Vaidya, Connor T. Jerzak, Nhat Ho, Chandrajit Bajaj
Queryable LoRA introduces a parameter-efficient fine-tuning method that dynamically routes over a shared memory of low-rank update atoms, guided by both the model's internal state and external language instructions. This approach consistently demonstrates improved generalization, enhanced optimization stability, and better distributed gradient flow across diverse tasks compared to static low-rank adaptation techniques.
Key insights
- Dynamic, context-sensitive low-rank adaptation can be achieved efficiently by routing over a bounded, shared set of update atoms, acting as a 'vocabulary' of transformations.
- Integrating language instructions provides a principled and bounded way to semantically bias model adaptation, balancing utility for the hidden state with alignment to the instruction.
- Sharing and reusing adaptation patterns across layers via blockwise routing improves gradient flow and promotes the learning of reusable update directions, particularly benefiting deeper architectures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.