Reproduce “Dynamics Reveals Structure: Challenging the Linear Propagation Assumption” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hoyeon Chang, Bálint Mucsányi, Seong Joon Oh

Analyzing the dynamics of first-order parameter updates, this research uncovers the geometric constraints necessary for machine learning models to maintain logical coherence when adapting relational knowledge. It reveals that while unary logical operations mandate tensor-factorized feature structures, systematic composition proves incompatible with negation equivariance under linear propagation assumptions.

Key insights

  • Empirical analysis showed that gradients of factual queries and their negations in LLMs (e.g., Qwen3, OLMo3) are strongly positively aligned (e.g., ~0.85 cosine similarity), contradicting the anti-alignment required for negation-consistent linear updates.
  • Systematic linear propagation of unary logical operations (negation, converse) geometrically necessitates tensor-factorized feature spaces that separate entity-pair context from relation content, with specific symmetric and antisymmetric properties for handling directionality.
  • A fundamental structural conflict exists: combining negation equivariance with an idempotent, bilinear conjunction operator (required for systematic composition) leads to a 'collapse' where all feature vectors must be zero, indicating an inherent incompatibility within the first-order update regime.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.