Reproduce “Cortex: Consolidating Context into Parametric Memory” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jintian Zhang, Feng He, Xinnian Liang, Ziwen Xu, Haoming Xu, Haojing Huang, Shuangzhi Wu, Ningyu Zhang
Cortex consolidates context-induced behavior into model weights so later answers can use it after the source instruction is removed. It distills a context-conditioned teacher into a student without the context, using new feed-forward parameters and replay across updates. On Qwen2.5-14B-Instruct after 50 ZsRE edits, Structural achieved 88.9% CR and 99.8% RR on specified evaluations, not all capabilities.
Key insights
- Memory Consolidation evaluates both recovery of context-induced gains (Consolidation Rate) and preservation of non-target performance (Retention Rate), treating them as distinct objectives.
- The best update design depends on the task: Structural updates tend to favor code learning and compositional portability, while Synaptic updates more often perform well on factual editing.
- Replay is important for continual updates: without it, performance on earlier facts can drop sharply, whereas replay keeps it above 96% after 25 later updates in the reported experiment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.