Reproduce “Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Flavio Di Palo, Prateek Singhi, Bilal Fadlallah
Amazon researchers developed Performance-Guided Knowledge Distillation (PGKD), an iterative active learning method for efficient multi-class text classification. This approach distills LLM knowledge into smaller models, achieving up to 130X faster and 25X less expensive inference compared to direct LLM use, while also improving classification accuracy, especially for datasets with many classes.
Key insights
- Providing continuous, performance-aware feedback (class-wise validation metrics) to the teacher LLM significantly enhances the student model's learning and adaptability, particularly in highly multi-class contexts.
- Explicitly focusing on hard negative samples—misclassified examples with high confidence—is crucial for refining the student model's decision boundaries and improving its robustness.
- The iterative data generation guided by student performance, combined with hard negative mining, allows LLMs to effectively generate training data that addresses specific student weaknesses, leading to substantial performance gains in complex classification tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.