Reproduce “Set the Clock: Temporal Alignment of Pretrained Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bowen Zhao, Zander Brumbaugh, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith
Research introduces methods to align the internal knowledge of pretrained language models to a specific temporal context, demonstrating that models can activate existing factual information for a target year without explicit temporal cues. A new dataset, TAQA, was created to facilitate the evaluation of these temporal alignment techniques.
Key insights
- Pretrained LMs exhibit strong temporal biases, with performance often peaking on historical data (e.g., LLaMa2-70B performed best on 2019 data despite a 2022 cutoff), confirming their chaotic temporal understanding.
- Temporal alignment primarily activates existing, implicitly held knowledge within the LM rather than injecting new facts, particularly when finetuning data is selected based on "model correctness."
- Popular questions and data selected for model correctness showed greater benefits from alignment, suggesting a link between knowledge memorization and alignability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.