Reproduce “GTFormer: A Syntax-Free Graph Network Alternative to Transformers for 8192$\times$ Length Extrapolation in Linear Time” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Elisabeth Fink
GTFormer replaces all-to-all token comparisons with local neighbor interactions followed by hierarchical tree reduction and per-channel convex gates. It evaluates the method on eight procedurally generated tasks, measuring joint exact-match accuracy. The best variant reaches 98.1% at 4,194,304 tokens, after training only up to 512; this result is limited to synthetic tasks, not real-world sequences.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.