Reproduce “Document clustering with evolved multiword search queries” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Laurence Hirsch, Robin Hirsch, Bayode Ogunleye
The `eSQ` system introduces an interpretable document clustering method utilizing a genetic algorithm to evolve multi-word search queries for cluster definition. This approach consistently outperformed k-means++ and other traditional algorithms in V-measure (0.730 vs. 0.628) and Adjusted Rand Index (0.728 vs. 0.525) across diverse datasets, providing human-readable explanations for cluster formation.
Key insights
- The "intersect requirement" (at least 50% document overlap for new query words) is crucial for generating semantically coherent multi-word queries and effective clusters.
- Applying a small penalty (e.g., `kPenalty` of 0.02) for increasing the number of clusters (`k`) in the GA's fitness function improves the discovery of true underlying categories.
- While KNN expansion successfully assigns all documents, it generally reduces V-measure, indicating the strong initial quality of the GA-generated query-based clusters prior to expansion.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.