Reproduce “Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiangchen Song, Zhenhao Chen, Lingjing Kong, Shaoan Xie, Xinshuai Dong, Guangyi Chen, Kun Zhang
Researchers from Carnegie Mellon University and MBZUAI developed a behavioral evaluation framework to calibrate claims about Test-Time Training (TTT) memory in large language models. This framework demonstrates that while TTT methods show improvements in proxy metrics like perplexity, these do not consistently translate to robust, behavioral recall of sparse facts after their supporting context is removed.
Key insights
- Improvements in proxy metrics like negative log likelihood (NLL) in Test-Time Training (TTT) do not automatically equate to robust, open-ended behavioral recall of learned information in deployment-like scenarios.
- Stronger TTT claims regarding deployment-time memory require specific behavioral evaluations covering delayed recall, paraphrase robustness, locality, and conflict resolution, which go beyond local likelihood gains.
- An "evidence migration" pattern is prevalent in current TTT literature, where S- and B-level evidence is frequently used to motivate D-level deployment narratives without adequate behavioral validation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.