Reproduce “NN-Former: Rethinking Graph Structure in Neural Architecture Representation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruihan Xu, Haokui Zhang, Yaowei Wang, Wei Zeng, Shiliang Zhang
Researchers at Peking University and collaborators introduced NN-Former, a neural predictor framework that refines neural architecture representation by explicitly modeling adjacency and previously overlooked sibling node relationships within a transformer-based architecture. This method consistently achieves superior accuracy and latency prediction across various benchmarks, notably improving out-of-domain latency prediction by an average of 11.61% in Acc(10%) compared to previous leading approaches.
Key insights
- Sibling nodes are a critical topological feature for understanding neural architectures, as they represent complementary features for accuracy and potential concurrent execution for latency.
- Explicitly modeling diverse local topological relationships through masked attention (ASMA) significantly improves feature representation and prediction accuracy in transformers.
- A bidirectional graph isomorphism feed-forward network (BGIFFN) can effectively capture directed structural information, making traditional position encodings redundant and enhancing the model's robustness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.