Reproduce “OOD Detection for EEG-based Machine Learning in High-Risk Environments” on OpenResearch

About this paper

Philipp Bomatter, Henry Gouk

An EEG-focused benchmark evaluates whether OOD detection methods can identify realistic preprocessing and acquisition shifts, and whether their scores improve clinical prediction safety. Generative approaches, especially SITN, consistently outperformed classifier-confidence methods for detecting shifted inputs, while MSP was more useful for identifying difficult in-distribution cases; combining both raised TUAB accuracy from 67.5% to 84.1% under mixed ID/OOD evaluation, at 43.7% coverage.

Key insights

• OOD detection and model uncertainty represent different safety problems: SITN separates unfamiliar inputs, whereas MSP more often identifies ambiguous in-distribution samples.
• Raw likelihood can fail because generative models may assign high likelihood to simple or smoothed signals that are nevertheless out of distribution; performance depends strongly on the type of shift.
• Reliable evaluation should use realistic covariate perturbations and test whether scores track degradation in the downstream clinical task, not only whether they separate datasets.

Summary by alphaXiv.