Reproduce “Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chirag Shah, Ryen W. White, Reid Andersen, Georg Buscher, Scott Counts, Sarkar Snigdha Sarathi Das, Ali Montazer, Sathish Manivannan and 8 more

Microsoft and university researchers introduced a human-in-the-loop methodology that uses large language models (LLMs) to generate, validate, and apply user intent taxonomies from log data. This approach demonstrated that LLMs can achieve high consistency and objectivity in annotations, often surpassing human agreement, while substantially reducing the effort for qualitative analysis.

Key insights

  • LLMs can reliably generate initial user intent taxonomies with stable top-level categories, especially when guided by iterative human refinement and improved prompt engineering.
  • LLMs, particularly GPT-4, can offer more objective and consistent annotations than human coders by avoiding over-extrapolation or subjective inferences.
  • The methodology demonstrates a paradigm where LLMs enhance human capabilities in qualitative data analysis, accelerating research and leading to more consistent outcomes beyond mere efficiency gains.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.