Reproduce “Latent Feedback Is a Compute and Data Amplifier: Scaling Law for Latent Feedback Transformers on Repeated Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Benhao Huang, Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno, Hector Liu
The authors test latent feedback, which passes each token’s top-layer state into the next token’s input, against a repeated-data Transformer. At matched validation loss, it uses 48% less compute (60M, 400 UTPP, two epochs) and 41% less unique data (271M, 100 UTPP, three epochs). It can lose when data is scarce; evidence covers one recipe, dataset and model ladder.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.