Reproduce “Neuro-BERT: Rethinking Masked Autoencoding for Self-supervised Neurological Pretraining” on OpenResearch

About this paper

Di Wu, Siyuan Li, Jie Yang, Mohamad Sawan

Researchers at Westlake University developed Neuro-BERT, a self-supervised pre-training framework that redefines masked autoencoding for neurological signals using a Fourier Inversion Prediction (FIP) task. This approach yields state-of-the-art performance on diverse downstream tasks like sleep stage classification and gesture recognition, demonstrating strong effectiveness with limited labeled data.

Key insights

• The frequency and phase characteristics of neurological signals are bio-inspired indicators of underlying activities, making them suitable targets for representation learning.
• Masked autoencoding, when re-imagined with Fourier domain prediction, effectively handles the informational sparsity and stochasticity of neurological signals, outperforming direct spatiotemporal reconstruction.
• Rich, generalizable representations can be learned by a transformer encoder through Fourier Inversion Prediction, allowing for a lightweight linear prediction head during pre-training.

Summary by alphaXiv.