Reproduce “Introducing Three New Benchmark Datasets for Hierarchical Text Classification” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jaco du Toit, Herman Redelinghuys, Marcel Dunaiski
Researchers from Stellenbosch University introduced three new, well-documented and balanced benchmark datasets (WOS_JT, WOS_CT, WOS_JTF) for Hierarchical Text Classification (HTC) of research publications, aiming to improve reproducibility and model evaluation. The WOS_JTF dataset, created by combining and filtering journal and citation-based classifications, demonstrated enhanced semantic cohesion and separability, leading to an increase of up to 17.34 Micro-F1 and 20.54 Macro-F1 for state-of-the-art HTC models compared to a purely journal-based dataset.
Key insights
- Combining journal-based and citation-based classification schemas with a filtering process, as implemented in WOS_JTF, improves the semantic cohesion of classes and separation between them.
- Comprehensive documentation of dataset creation methodologies is critical for advancing HTC research by ensuring reproducibility and enabling reliable comparisons between different classification models.
- Balanced datasets with clear class boundaries lead to superior performance for machine learning-based HTC models, highlighting the importance of dataset design over merely increasing data volume.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.