Reproduce “Multi-Task Pre-Finetuning of Lightweight Transformer Encoders for Text Classification and NER” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junyi Zhu, Savas Ozkan, Andrea Maracani, Sinan Mutlu, Cho Jung Min, Mete Ozay
Researchers at Samsung R&D Institute UK developed Multi-Task Pre-Finetuning with Task-Primary LoRAs (MTPF-TPL), a framework for lightweight transformer encoders to simultaneously optimize for named entity recognition (NER) and text classification (TC) tasks while mitigating task interference. This approach achieves performance comparable to individually pre-finetuned models, yielding gains such as +0.7% F1 for NER and +8.8% accuracy for TC on MiniLM, supporting efficient deployment on mobile platforms.
Key insights
- Individual pre-finetuning objectives for NER and text classification create conflicting representational requirements in a shared encoder, leading to performance degradation on the opposing task.
- NER pre-finetuning reduces sentence-level distinctiveness crucial for text classification, while text classification pre-finetuning increases intra-sentence token homogeneity, harming NER.
- Task-Primary LoRAs (TPLs) effectively reconcile these conflicts by localizing task-specific adaptations to dedicated LoRA modules within the shared backbone, allowing joint optimization without severe interference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.