Reproduce “DISCO: Distilling Counterfactuals with Large Language Models” on OpenResearch

About this paper

Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, Kyle Richardson

The research introduces DISCO, an automated pipeline that leverages large language models (LLMs) to generate diverse counterfactual data, which is then refined by a task-specific teacher model. This approach scales the creation of high-quality counterfactuals, leading to improved robustness and out-of-distribution generalization for NLP models, particularly in Natural Language Inference.

Key insights

• Large Language Models can effectively produce diverse and semantically complex candidate counterfactuals that go beyond simple negations.
• Combining LLM overgeneration with an automatic, task-specific teacher model for filtering enables the production of high-quality counterfactual data at scale without human annotation.
• Targeted, high-quality counterfactual data, even in relatively smaller quantities, can be more effective for enhancing model robustness and generalization than much larger, less precisely generated datasets.

Summary by alphaXiv.