Reproduce “AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu and 2 more
Researchers from The University of Hong Kong and Microsoft Research Asia developed AnnoLLM, an LLM-powered system that functions as a data annotator and generator using an "explain-then-annotate" strategy. This method enabled large language models to surpass human crowdsourcers on one task and match human performance on another, while also producing a high-quality dataset for conversational information retrieval.
Key insights
- LLMs can consistently produce high-quality, relevant explanations for labels when guided by ground truth, significantly enhancing their annotation capabilities.
- The "explain-then-annotate" strategy is more effective than standard few-shot prompting, improving accuracy and leveraging LLMs' reasoning potential in data annotation.
- AnnoLLM's CoT prompts enhance annotation stability, making the LLM less sensitive to minor variations in prompt templates and providing more robust performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.