Reproduce “Symbolic Knowledge Distillation: from General Language Models to Commonsense Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck and 1 more
Researchers at the University of Washington and AI2 developed Symbolic Knowledge Distillation, a framework that leverages large language models to automatically generate and filter high-quality commonsense knowledge graphs, surpassing human-authored versions in scale and quality. This process enables training more compact, specialized models that outperform the larger generative models and human-trained baselines in commonsense inference.
Key insights
- Large general language models (like GPT-3) can serve as effective "knowledge generators" for structured, symbolic commonsense information, not just unstructured text.
- A small amount of human-labeled data can be effectively leveraged to train a "critic" model to significantly improve the quality of machine-generated symbolic knowledge.
- Training a smaller, specialized "student" model on a large, high-quality machine-generated symbolic corpus can lead to performance superior to both its much larger general-purpose "teacher" and models trained on human-authored data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.