Reproduce “Automated Thematic Analyses Using LLMs: Xylazine Wound Management Social Media Chatter Use Case” on OpenResearch

About this paper

JaMor Hairston, Ritvik Ranjan, Sahithi Lakamana, Anthony Spadaro, Selen Bozkurt, Jeanmarie Perrone, Abeed Sarker

Researchers at Emory University and collaborating institutions demonstrated that Large Language Models can effectively replicate expert-driven thematic analysis of social media discussions about xylazine wound management. The study found that framing the task as multiple binary classifications and using few-shot prompting enabled LLMs to achieve over 90% accuracy in aligning with human expert classifications, providing a scalable tool for public health surveillance.

Key insights

• Decomposing thematic analysis into a series of multiple binary classification tasks significantly improves Large Language Model (LLM) performance over a single multi-label approach.
• Few-shot prompting, particularly two-shot examples, is critical for guiding LLMs to accurately detect themes, align with expert coding, and overcome issues like overfitting observed in zero-shot approaches.
• LLMs can effectively track temporal shifts in online discourse related to public health issues, demonstrating their utility for dynamic surveillance.

Summary by alphaXiv.