Reproduce “NAR-Former V2: Rethinking Transformer for Universal Neural Network Representation Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yun Yi, Haokui Zhang, Rong Xiao, Nannan Wang, Xiaoyu Wang
NAR-Former V2 combines Transformer-based global modeling with graph-constrained attention to represent both compact NAS cells and complete, deeply connected neural networks. On latency prediction, it reduces average MAPE to 3.07% in same-distribution testing and to 10.55% when an entire network family is held out, while accuracy-ranking results remain competitive with the original NAR-Former.
Key insights
- Topology-constrained attention improves the Transformer’s inductive bias for complete neural networks: on EfficientNet, graph-aided attention lowers MAPE from 16.88% to 13.20% and raises Acc(10%) from 36.32% to 44.01% relative to unrestricted attention.
- Connectivity itself carries useful semantic information about operations such as branching and concatenation, and the type-aware module provides additional gains beyond graph attention alone.
- The method’s advantage is clearest in cross-family generalization for latency prediction; its accuracy-prediction improvements over the original NAR-Former are mixed and depend on dataset size.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.