Reproduce “Names Don't Matter: Symbol-Invariant Transformer for Open-Vocabulary Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
İlker Işık, Wenchao Li
A Symbol-Invariant Transformer is introduced to achieve provable alpha-renaming invariance for interchangeable tokens in open-vocabulary learning. This architecture consistently attains perfect alpha-covariance (1.0) and demonstrates improved performance over baselines, including GPT-5.2, in out-of-distribution symbolic reasoning tasks such as propositional logic and Linear Temporal Logic witness generation.
Key insights
- Provable alpha-renaming invariance in Transformers can be achieved by structurally designing parallel processing streams with shared, commutative operations, eliminating reliance on stochasticity.
- Per-stream cross-attention is crucial for the decoder to maintain distinct information about each interchangeable token's context, significantly impacting output generation accuracy.
- Aggregated attention is vital for relational understanding among interchangeable tokens, though its specific impact varies with task demands, being more critical for propositional logic than for LTL.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.