Reproduce “Uncertainty Estimation for the Open-Set Text Classification systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Leonid Erlygin, Alexey Zaytsev
A Bayesian uncertainty estimator for open-set text classification combines ambiguity from the known-class gallery with reliability estimates for individual text embeddings. Across authorship, intent, and topic benchmarks, HolUE generally ranked recognition errors more effectively than threshold-, embedding-, or gallery-only baselines, reaching PRRs of 0.79 on Yahoo Answers, 0.95 on DBPedia, 0.73 on CLINC150 at low FPIR, and up to 0.55 on PAN authorship attribution.
Key insights
- Distance from an acceptance threshold is an incomplete uncertainty signal: it can miss errors caused by overlapping known classes or ambiguous sample representations.
- The relative importance of uncertainty sources depends on the task. Gallery structure is especially useful for authorship attribution, while combining gallery and embedding information is consistently more effective across the evaluated settings.
- Evaluating uncertainty through Prediction Rejection Ratio directly measures whether uncertain samples are the ones most likely to produce operational classification errors, rather than only measuring aggregate OOD detection.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.