Reproduce “A Fully Automated Pipeline for Conversational Discourse Annotation: Tree Scheme Generation and Labeling with Large Language Models” on OpenResearch

About this paper

Kseniia Petukhova, Ekaterina Kochmar

Researchers developed a fully automated pipeline that uses Large Language Models (LLMs) to generate hierarchical annotation schemes and then label conversational discourse. This system, especially when employing frequency-guided tree creation and GPT-4o for annotation, outperformed human crowdsourcers using manually designed schemes and reduced annotation time from 31 hours to 1 hour 25 minutes, achieving 98% accuracy at the first tree level.

Key insights

• Allowing LLMs to generate open-ended questions and non-binary splits significantly improves the effectiveness of automatically constructed annotation trees.
• Incorporating class frequency information into the tree generation process, guiding LLMs to prioritize frequent classes, notably enhances overall annotation accuracy and reduces root-level errors.
• Advanced LLMs like GPT-4o are capable of not only following complex annotation guidelines but also autonomously generating high-quality hierarchical schemes themselves.

Summary by alphaXiv.