Reproduce “When Models Don't Manipulate Manifolds: The Geometry of a Comparison Task” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sai Sumedh R. Hindupur, Hadas Orgad, Thomas Fel, Demba Ba
The authors investigate how Qwen2.5-7B-Instruct compares two-digit numbers in a max-completion task. They identify linear directions carrying each number, which attention brings together at the second number’s position, where MLP neurons compare them. In held-out counterfactuals perturbing the second number, patching the shared plane yields 0.940 whole-number IIA, versus 0.115 for the full layer-14 residual. This supports sufficiency, not necessity.
Key insights
- Curved activation geometry can coexist with linear features that causally support a specific computation; the model need not manipulate the full number manifold to compare values.
- For two numbers, an attention head transfers information about the first operand to the second operand’s position, where the two number features form a comparison-relevant representation.
- MLP neurons appear to divide the work between localized comparisons and later aggregation; for three numbers, the proposed process uses intermediate maximum flags, though the final readout mechanism remains unresolved.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.