Reproduce “On Limitation of Transformer for Learning HMMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiachen Hu, Qinghua Liu, Chi Jin

Research from Peking and Princeton Universities investigates Transformer performance on Hidden Markov Models (HMMs), showing RNNs consistently outperform Transformers in training speed and accuracy across various HMM types. Transformers demonstrate a logarithmic depth scaling for structured HMMs, struggle with instances requiring complex hidden state inference from sparse observations, and benefit from integrated recursive biases like 'block Chain-of-Thought' to handle longer sequences.

Key insights

  • Transformers lack the natural recursive inductive bias of RNNs, making them less efficient and robust for precise, step-by-step sequential computations inherent in HMMs.
  • While Transformers excel at capturing long-range dependencies, this advantage does not universally translate to superior performance on all sequential tasks; they are consistently outperformed by RNNs on fundamental HMMs.
  • Integrating explicit recursive mechanisms, such as 'block Chain-of-Thought,' can significantly enhance Transformer performance on long sequence dependencies by providing a structured inductive bias.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.