Reproduce “Promoting CNNs with Cross-Architecture Knowledge Distillation for Efficient Monocular Depth Estimation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhimeng Zheng, Tao Huang, Gongsheng Li, Zuyi Wang
DisDepth introduces a method to enhance monocular depth estimation by effectively distilling knowledge from high-performing transformer models into computationally efficient CNN architectures. The approach achieves competitive accuracy on KITTI and NYU Depth V2 datasets while significantly reducing FLOPs, parameters, and latency, making it suitable for resource-constrained environments.
Key insights
- Effective cross-architecture knowledge distillation from a transformer to a CNN requires an 'acclimation' step to decouple task-relevant 'extrinsic information' from 'architecture-intrinsic information' in the teacher's features.
- Simple, convolution-based Local-Global Convolution modules can efficiently imbue CNNs with improved global contextual understanding without resorting to complex transformer blocks or requiring full backbone retraining.
- Guiding the student's distillation loss with attention mechanisms, focusing on regions deemed important by the teacher, enhances the transfer of valuable knowledge and improves overall student performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.