Reproduce “Learning on Model Weights using Tree Experts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Eliahu Horwitz, Bar Cavia, Jonathan Kahana, Yedid Hoshen
Researchers at The Hebrew University of Jerusalem developed ProbeX, an efficient method for inferring neural network functionality directly from their weights. The approach leverages the structure of 'Model Trees' and enables zero-shot classification and retrieval of models by aligning their weights with semantic language representations, significantly improving model discoverability on platforms like Hugging Face.
Key insights
- Learning on model weights is significantly simpler when models share a common initialization point, forming 'Model Trees', due to reduced nuisance variation.
- A lightweight 'single-layer probing' architecture can effectively capture internal model representations while remaining computationally efficient for large models.
- Aligning model weights with external semantic representations (like text embeddings) enables powerful zero-shot classification and retrieval capabilities for models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.