Reproduce “Unsupervised Out-of-Domain Detection via Pre-trained Transformers” on OpenResearch

About this paper

Keyang Xu, Tongzheng Ren, Shikun Zhang, Yihao Feng, Caiming Xiong

Researchers from Columbia University, UT Austin, CMU, and Salesforce Research introduce an unsupervised method for out-of-domain (OOD) text detection that leverages all layers of pre-trained transformers via Mahalanobis Distance as Features (MDF) and unsupervised fine-tuning. This approach achieved up to a 14.0% absolute AUROC improvement over single-layer baselines, enabling robust OOD detection without requiring task-specific labels.

Key insights

• No single transformer layer consistently yields optimal OOD detection; the best layer varies significantly with dataset and OOD type, underscoring the need for a multi-layer approach.
• Using Mahalanobis distance to combine features from all transformer layers (MDF) effectively captures the in-domain distribution and feature correlations, substantially outperforming single-layer or simple aggregation methods.
• Unsupervised fine-tuning of pre-trained transformers through methods like In-domain Masked Language Modeling (IMLM) or Binary Classification with Auxiliary Dataset (BCAD) significantly improves the discriminative power of features for OOD detection.

Summary by alphaXiv.