Reproduce “Circuits, Features, and Heuristics in Molecular Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kristof Varadi, Mark Marosi, Peter Antal

This research dissects an autoregressive molecular transformer to reveal the computational structures it uses to generate valid chemical compounds. It identifies specific attention heads responsible for enforcing SMILES grammar and demonstrates that abstract chemical concepts like valence are linearly decodable and causally linked to bond predictions. Using sparse autoencoders, the study extracted interpretable chemical features that significantly enhance molecular property prediction and enable controlled molecular generation.

Key insights

  • Molecular transformers develop distinct computational circuits for handling SMILES grammar, with specific attention heads acting as state trackers and others as causal executors for syntactic constraints.
  • Fundamental chemical concepts like valence capacity are robustly encoded and linearly decodable within the transformer's residual stream, exhibiting a causal influence on the model's bond-order predictions.
  • Sparse Autoencoders effectively disentangle polysemantic neuron activations into monosemantic, chemically interpretable features that are robust across various transformations and universal across different SAE instances.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.