Reproduce “Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Haodong Wen, Yiran Zhang, Yingfa Chen, Kaifeng Lyu

Frontier large language models struggle with exact string copying, a seemingly basic task, particularly with increasing input length or repetitions. Researchers from Tsinghua University introduce 2D-RoPE, a novel positional encoding that views text as a 2D grid, leading to perfect copying for input lengths hundreds of times greater than training data and maintaining comparable general language understanding.

Key insights

  • Despite advanced reasoning, frontier LLMs lack fundamental exact copying abilities due to the inductive biases of their 1D positional encodings.
  • 1D shift-invariant positional encodings are theoretically limited in expressing exact copy rules, requiring ad-hoc learning mechanisms that do not generalize.
  • Structuring text as a 2D grid with corresponding positional encodings (2D-RoPE) provides a more direct and theoretically provable mechanism for length-generalizing exact copying.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.