Reproduce “Automating Learner Assessment: Benchmarking Machine Learning and Deep Learning Models for EEG-Based Familiarity Prediction” on OpenResearch

About this paper

Isuru Nanayakkara, Thilina Halloluwa

This study benchmarks machine learning and deep learning models for predicting learner familiarity from EEG signals, assessing performance across stimuli like faces and mathematical equations. Researchers found that while models initially showed high accuracy (e.g., CNN 98.53% F1-score) with standard validation, a more rigorous trial-independent protocol revealed a realistic performance of 60.38% F1-score, establishing a robust benchmark. High Gamma and Low Gamma bands in temporal and parietal regions were identified as critical neurophysiological biomarkers for familiarity detection.

Key insights

• The validation methodology is crucial; standard cross-validation can inflate performance substantially, as demonstrated by the CNN's F1-score dropping from 98.53% to 60.38% under trial-independent validation.
• Despite the performance adjustment, models significantly outperformed chance levels, confirming the feasibility of objectively decoding cognitive familiarity from EEG signals.
• High Gamma (32.2%) and Low Gamma (26.2%) bands, particularly in temporal and parietal channels (T3, P3, F7, T4), are key neurophysiological features for familiarity prediction.

Summary by alphaXiv.