Reproduce “QuaLLM: An LLM-based Framework to Extract Quantitative Insights from Online Forums” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Varun Nagaraj Rao, Eesha Agarwal, Samantha Dalal, Dan Calacci, Andrés Monroy-Hernández

QuaLLM, an LLM-based framework, introduces a structured methodology for thematic analysis and quantitative insight extraction from online forums. Its application to over one million Reddit comments from rideshare workers successfully surfaced dominant concerns related to algorithmic transparency, predictability, and fairness, demonstrating scalable interpretive analysis.

Key insights

  • LLMs can effectively mimic and scale traditional qualitative research steps like 'open coding' and 'thematic analysis', allowing for the systematic extraction of interpretive, quantitative insights from vast unstructured text datasets.
  • The human-LLM inter-rater reliability values for classification and prevalence tasks were very close to human-human values, suggesting that LLMs can reliably substitute human labels for these specific analytical steps.
  • The framework provides a financially feasible approach ($1,662.30 for over one million comments) for conducting large-scale thematic analysis, bridging the gap between qualitative depth and quantitative breadth.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.