Reproduce “AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures” on OpenResearch

About this paper

Michael S. Ryoo, AJ Piergiovanni, Mingxing Tan, Anelia Angelova

AssembleNet introduces a neural architecture search (NAS) method to automatically discover multi-stream, multi-resolution neural connectivity patterns for video architectures. It achieves state-of-the-art performance on video understanding benchmarks like Charades and Moments-in-Time datasets by efficiently exploring the complex design space.

Key insights

• The connection-learning-guided evolution significantly outperforms traditional random mutation and purely random search, demonstrating that leveraging learned connection weights effectively prunes the search space and accelerates discovery of high-performing architectures.
• Automated search discovers superior connectivity patterns compared to a range of hand-designed multi-stream (2+1)D CNN models, highlighting the limitations of human intuition for complex multi-modal architectures.
• The discovered architectures frequently utilize multiple blocks with varying temporal resolutions at the same level and often prefer more than two input streams, suggesting the benefits of multi-resolution processing and richer multi-modal inputs.

Summary by alphaXiv.