Reproduce “Toward Purpose-oriented Topic Model Evaluation enabled by Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhiyin Tan, Jennifer D'Souza

Researchers at L3S Research Center and TIB Leibniz Information Centre developed a unified framework that utilizes Large Language Models to evaluate topic model quality across four dimensions and nine metrics. The framework, validated with adversarial tests, provides fine-grained diagnostic insights into topic model performance, revealing that traditional coherence metrics often fail to penalize redundancy.

Key insights

  • LLM-based metrics, especially `AdvT_nonword` and `AdvT_outlier`, demonstrated strong diagnostic reliability in adversarial tests, averaging over 90% and 88% accuracy, respectively.
  • An antagonistic relationship exists between LLM-judged topic coherence and repetitiveness, indicating that traditional coherence metrics often fail to penalize or may even reward redundancy.
  • Different LLM families exhibit unique "evaluator profiles" in terms of strictness and sensitivity, with larger LLMs generally performing better but also showing resource-efficient alternatives among base models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.