Reproduce “Enabling Efficient Hardware Acceleration of Hybrid Vision Transformer (ViT) Networks at the Edge” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Joren Dumoulin, Pouya Houshmand, Vikram Jain, Marian Verhelst
Researchers from KU Leuven developed a hardware accelerator and co-optimization strategy for Hybrid Vision Transformer (ViT) networks, significantly improving their efficiency on edge devices. Their design enables high throughput and low power consumption by addressing diverse layer types and reducing memory transfers for intermediate data and non-linear operations, achieving 13.16 FPS for EdgeNeXt-S at 18.4 mW.
Key insights
- Hardware-scheduling co-optimization across spatial, temporal, and inter-layer dimensions is crucial for efficiently accelerating complex, heterogeneous workloads like hybrid ViTs.
- Minimal hardware overhead (1.1%) is sufficient to introduce dynamic reconfigurability for diverse layer support, yielding substantial performance gains.
- Fusing non-linear operations and intermediate layer computations can significantly mitigate memory bottlenecks, especially for structures like inverted bottlenecks that are prevalent in efficient edge models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.