Reproduce “Towards Efficient and Explainable Hate Speech Detection via Model Distillation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Paloma Piot, Javier Parapar

Researchers from IRLab, CITIC, Universidade da Coruña developed a knowledge distillation method that transfers hate speech detection and explanation capabilities from a 70B-parameter LLM to an 8B-parameter model. The distilled model achieved an F1-score of 0.85, surpassing its larger teacher by 9%, and generated explanations with 91% correctness, nearly matching the teacher, while being 54% faster and requiring 80% less GPU memory.

Key insights

  • Knowledge distillation effectively transfers both classification accuracy and natural language explanation generation abilities from a large LLM to a significantly smaller, more efficient model.
  • Fine-tuning a smaller model with targeted, high-quality data and a multi-task objective can enable it to outperform a much larger teacher model for specific downstream tasks like hate speech detection and explanation.
  • High-quality, human-understandable explanations for hate speech can be generated by efficient, distilled models, addressing transparency needs without incurring prohibitive computational costs or resource requirements.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.