Reproduce “Multi-objective Differentiable Neural Architecture Search” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler, Samuel Dooley, Josif Grabocka, Frank Hutter
Research from the University of Freiburg introduces Multi-Objective Differentiable Neural Architecture Search (MODNAS), an algorithm that generates diverse Pareto-optimal neural architectures balancing performance and hardware efficiency across multiple devices in a single search run. It significantly reduces computational cost from O(T) or O(C) to O(1) by leveraging a meta-hypernetwork, achieving superior Hypervolume indicators on NAS-Bench-201 and extending to Transformer architecture search.
Key insights
- A single, meta-learned hypernetwork can effectively generalize across various hardware devices and user preferences to efficiently generate diverse Pareto-optimal architectures in one search run.
- Multiple Gradient Descent (MGD) is critical for robustly optimizing the MetaHypernetwork by computing a common gradient direction that aims to simultaneously improve all device-specific objectives.
- The MetaHypernetwork learns device embeddings that cluster similar hardware characteristics, enabling zero-shot transferability for generating Pareto-optimal architectures for previously unseen devices.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.