Reproduce “TPTT: Transforming Pretrained Transformers into Titans” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fabien Furfaro
The TPTT framework enables upgrading existing pretrained Transformer language models with efficient linearized attention and memory mechanisms, mitigating quadratic complexity for long contexts. This approach, which uses parameter-efficient fine-tuning, demonstrated improvements such as increasing the MMLU Exact Match score for a Llama-3.2-1B variant from 0.007 to 0.246.
Key insights
- The TPTT framework successfully integrates efficient attention and memory augmentation into existing LLMs using PEFT, preserving pretrained knowledge with minimal architectural changes.
- Increasing the number of Householder updates ($n_h=2$) and employing derivative-based virtual token expansion (∆) in DeltaProduct enhances learning efficiency and convergence speed.
- The framework exhibits stable training dynamics and effective optimization across diverse Transformer architectures (e.g., Llama, OpenELM, Qwen) and various model sizes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.