Reproduce “Beyond Linear Concepts: Discovering and Aligning Non-Linear Concept Manifolds in Large Language Models” on OpenResearch

About this paper

Tido Specht, Elias Benedict Krey, Nils Neukirch, Nils Strodthoff

The authors adapt concept-based alignment to compare representations across layers, model sizes, model families, languages, and training stages. They cluster residual-stream activations into fuzzy concepts, then compare pairwise token-membership patterns across aligned observations without matching model coordinates. Qwen3 models show depth-structured correspondence, while matched-scale Llama–Qwen alignment is weaker. This provides finer-grained comparison, not evidence of causal concepts or universal metric superiority.

Key insights

• Concept coverage and clustering quality generally decline with depth, while layer alignment reveals diagonal and, in some models, intermediate- and late-layer block structures.
• Models within the tested Qwen family align more strongly than the compared Llama and Qwen models; English–Mandarin concept sharing also varies across model families.
• In the Tulu-3 case study, the largest measured representational change occurs from base to SFT, while DPO and RLVR remain closely aligned; alignment alone does not identify concepts’ causal roles.

Summary by alphaXiv.