Reproduce “Escaping the Big Data Paradigm with Compact Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, Humphrey Shi

The SHI Lab @ University of Oregon and collaborators introduced Compact Transformers (CCT), demonstrating that Transformer-based models can be effectively trained from scratch on small datasets, achieving competitive or superior performance to state-of-the-art CNNs. CCT models incorporate convolutional inductive biases and a novel sequence pooling strategy, enabling high accuracy with significantly reduced computational and parameter footprints.

Key insights

  • Injecting CNN-like inductive biases, particularly through a convolutional tokenizer, is crucial for enabling Transformer models to generalize effectively on small datasets from scratch.
  • Attention-based sequence pooling can more effectively aggregate information from Transformer output tokens compared to relying solely on a single [class] token.
  • Convolutional tokenizers can sufficiently encode spatial information, reducing or eliminating the need for explicit positional embeddings in Transformer models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.