Reproduce “WASD: Wasserstein-based Knowledge Distillation for Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Byeonghu Na, Donghyeok Shin, Yeongmin Kim, Mina Kang, Il-Chul Moon
WASD distills a teacher’s next-token predictions into a smaller student by using distances between token embeddings, so probability shifts between related tokens cost less than shifts between unrelated tokens. It applies debiased Sinkhorn divergence through entropy-regularized optimal transport and assumes a shared vocabulary. In a controlled GPT-2 XL-to-Base comparison, WASD scores 22.47 ROUGE-L versus CSD’s 21.46, indicating reference-text overlap, not guaranteed human-judged quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.