Reproduce “Structure-based Anomaly Detection and Clustering” on OpenResearch

About this paper

Filippo Leveni

Filippo Leveni's doctoral dissertation introduces unsupervised methodologies for structure-based anomaly detection and clustering, alongside advancements in density-based online anomaly detection and open-set recognition for malware discovery. The work demonstrates improved performance and efficiency across diverse datasets by leveraging structural priors and adaptive streaming algorithms.

Key insights

• Embedding data into a 'Preference Space' based on its fit to various parametric models effectively highlights deviations from expected structures, making anomalies more isolatable than in the ambient space.
• Integrating model fitting and selection (using GRIC) directly into an agglomerative clustering process allows for robust recovery of multiple, non-nested structures amidst noise, overcoming limitations of prior preference-based methods.
• Dynamic bin splitting and merging in a multi-resolution histogram within decision trees enables continuous and efficient adaptation to evolving data distributions in a streaming context without requiring periodic retraining.
• Leveraging the decision values ('LOGITS') from tree-based classifiers, combined with thresholding, provides an efficient and effective open-set recognition mechanism suitable for high-dimensional binary data in cybersecurity applications.

Summary by alphaXiv.