Reproduce “Learning with Less: Knowledge Distillation from Large Language Models via Unlabeled Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Juanhui Li, Sreyashi Nag, Hui Liu, Xianfeng Tang, Sheikh Sarwar, Limeng Cui, Hansu Gu, Suhang Wang and 2 more
LLKD distills an instruction-tuned Llama-3 teacher into a smaller RoBERTa student using only selected pseudo-labeled examples from an unlabeled corpus. By retaining samples that combine high teacher confidence with high student uncertainty, the method improves text-classification performance while using as little as 3.7% of the available training exposure on PubMed-RCT-20k; its weighted variant reaches the best overall average rank across five benchmarks.
Key insights
- Reliable pseudo-labels and informative examples are complementary selection criteria: ablations show that removing either teacher confidence or student uncertainty lowers performance.
- Adaptive thresholds outperform a single fixed cutoff by accounting for batch-level behavior and differences among classes.
- The student can surpass direct few-shot teacher predictions, indicating that carefully selected pseudo-labeled data can convert broad LLM knowledge into a more efficient task-specific model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.