Reproduce “Cluster Analysis and Concept Drift Detection in Malware” on OpenResearch

About this paper

Aniket Mishra, Mark Stamp

Researchers at San Jose State University developed an unsupervised clustering method, utilizing MiniBatch K-Means and silhouette coefficients, to detect concept drift in malware data. This approach allows machine learning models to adaptively retrain, achieving classification accuracies within 1% of continuous periodic retraining while reducing model updates by approximately 40%.

Key insights

• The inherent and continuous evolution of malware leads to concept drift, rendering static machine learning detection models increasingly inaccurate over time.
• Unsupervised clustering, specifically MiniBatch K-Means combined with the silhouette coefficient, can effectively identify and quantify shifts in malware data distributions, indicating concept drift.
• Adaptive model retraining, triggered only upon detected concept drift, can maintain high classification accuracy comparable to frequent periodic retraining while significantly reducing computational overhead.

Summary by alphaXiv.