Reproduce “Kalman Delta Networks: Uncertainty-aware Associative Memory” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ngoc Bui, Tinglin Huang, Rex Ying

Kalman Delta Networks (KDN) derive associative-memory write strengths from estimated uncertainty, rather than relying only on token-predicted gates. Diagonal and isotropic approximations make this idea compatible with parallel scans; in tests at 750M and 1.3B parameters, they improve several language-modeling, reasoning, and retrieval results over recurrent baselines, with different variants leading on different task groups.

Key insights

  • Uncertainty-aware gains offer a probabilistic interpretation of residual-based delta-rule updates: observations should influence memory according to both their residual and the confidence in the relevant memory direction.
  • The diagonal approximation preserves the exact one-step posterior mean under its retained predictive covariance, while approximating the posterior covariance; the isotropic version uses less state but retains less directional information.
  • The two approximations have different trade-offs: diagonal KDN is stronger on several retrieval evaluations, while isotropic KDN tracks baseline throughput closely and performs well in hybrid retrieval settings.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.