Reproduce “Transformers are Short Text Classifiers: A Study of Inductive Short Text Classifiers on Benchmarks and Real-world Datasets” on OpenResearch

About this paper

Fabian Karl, Ansgar Scherp

The authors test whether pretrained Transformers fine-tuned on labeled examples can classify short text without specialized short-text architectures. Across six benchmarks, Transformers lead on five; SearchSnippets is the exception, with ESGNN’s reported lead based on a modified split and extra preprocessing. On MR, DeBERTa reaches 90.21% accuracy versus 76.48% for WideMLP, while NICE-45 remains difficult.

Key insights

• General-purpose Transformer models, when properly fine-tuned, consistently outperform specialized short text classifiers and other deep learning models on various short text classification benchmarks.
• Claims of specialized models outperforming Transformers in prior work may stem from suboptimal Transformer tuning rather than inherent architectural limitations, highlighting the importance of thorough hyperparameter optimization.
• The inherent sparsity of short texts does not necessarily necessitate highly specialized model architectures; powerful pre-trained Transformers can effectively handle these challenges through their robust contextual understanding.

Summary by alphaXiv.