Reproduce “Cross-Architecture Knowledge Distillation (KD) for Retinal Fundus Image Anomaly Detection on NVIDIA Jetson Nano” on OpenResearch

About this paper

Berk Yilmaz, Aniruddh Aiyengar

Researchers at Columbia University developed a cross-architecture knowledge distillation framework that transfers high diagnostic accuracy from Vision Transformers to lightweight Convolutional Neural Networks for retinal fundus image anomaly detection. This framework achieved 89% accuracy on an NVIDIA Jetson Nano, retaining 93% of the teacher model's diagnostic performance while reducing its parameters by 97.4%.

Key insights

• Effective knowledge transfer between architecturally dissimilar models like ViTs and CNNs is achievable through purpose-built projection layers that align their disparate feature spaces and attention patterns.
• Pre-training the teacher Vision Transformer with I-JEPA self-supervised learning on unlabeled data significantly improved its ability to learn robust, semantically meaningful representations, which is crucial for medical imaging tasks with limited labeled datasets.
• A lightweight CNN student model can retain a high percentage (93%) of a high-capacity teacher's diagnostic accuracy, despite a substantial reduction in parameters (97.4%), making advanced AI accessible on edge devices.

Summary by alphaXiv.