Reproduce “Commonsense Generation and Evaluation for Dialogue Systems using Large Language Models” on OpenResearch

About this paper

Marcos Estecha-Garitagoitia, Chen Zhang, Mario Rodríguez-Cantelar, Luis Fernando D'Haro

Researchers from Universidad Politécnica de Madrid and National University of Singapore propose using Large Language Models to automatically generate and evaluate commonsense-infused dialogue turns for data augmentation. Their method leverages LLMs for zero-shot generation of 12 ATOMIC commonsense relations and evaluates quality via prompt-based classification, showing GPT-4's superior evaluation capabilities with an MRR up to 0.415, notably outperforming traditional methods.

Key insights

• Large Language Models (LLMs) can effectively generate diverse and contextually appropriate commonsense-rich dialogue turns for data augmentation.
• LLMs, particularly GPT-4, are highly effective as zero-shot evaluators for commonsense quality in dialogue, significantly outperforming traditional methods like ACCENT.
• While GPT-4 is optimal for evaluation, GPT-3.5 can be a cost-effective option for large-scale commonsense data generation without a drastic quality drop.

Summary by alphaXiv.