Reproduce “Systematic Generalization with Edge Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Leon Bergen, Timothy J. O'Donnell, Dzmitry Bahdanau
The Edge Transformer enhances systematic generalization in neural models by explicitly representing relations with edge states and employing a triangular attention mechanism for updates. It achieved state-of-the-art performance on compositional reasoning benchmarks like CLUTRR, CFQ, and COGS, outperforming existing Transformer and GNN baselines by notable margins.
Key insights
- Explicitly representing and dynamically updating relational information via edge states improves compositional generalization in neural models.
- The unification-inspired triangular attention mechanism provides a neural way to approximate rule-like computations for relational reasoning.
- Weight tying across layers, similar to Universal Transformers, can be beneficial for the Edge Transformer's performance on compositional tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.