Reproduce “Once-for-All: Train One Network and Specialize it for Efficient Deployment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, Song Han
Once-for-All (OFA) trains one flexible, shared-weight network that can yield smaller subnetworks for different device constraints. It adjusts depth, width, kernel size, and input resolution, then uses predictors and hardware-specific latency estimates to select a subnet. With 75 epochs of subnet fine-tuning, OFA reaches 80.0% ImageNet top-1 accuracy at 595M MACs, below 600M; this is not a no-fine-tuning result.
Key insights
- A single, universal network can be trained to contain an extremely large number of specialized sub-networks, provided a structured training approach like Progressive Shrinking is used to manage weight sharing and maintain performance.
- Decoupling the computationally intensive model training from the rapid, hardware-aware architecture search amortizes the cost, making deep learning deployment scalable and environmentally sustainable.
- Optimal neural network architectures are highly hardware-dependent, and an elastic universal model can be specialized to automatically discover these hardware-specific designs (e.g., kernel sizes for FPGAs vs. CPUs).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.