Reproduce “SLIM: Sparse Latent Steering for Interpretable and Property-Directed LLM-Based Molecular Editing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mingxu Zhang, Yuhan Li, Lujundong Li, Dazhong Shen, Hui Xiong, Ying Sun
The SLIM framework provides a plug-and-play method for precise, training-free control over molecular properties in large language model (LLM) editors. It achieves this by using a novel task-oriented Sparse Autoencoder to decompose LLM hidden states into sparse, interpretable, property-aligned features and applying gradient-based steering during inference, resulting in improved property editing success rates.
Key insights
- LLM hidden states can be effectively decomposed into sparse, interpretable features that are explicitly aligned with specific molecular properties, enabling more precise and controllable generation.
- A gradient-alignment loss is critical for training Sparse Autoencoders to learn feature representations that faithfully preserve causal steering directions, which is essential for effective property control, particularly for discrete properties.
- Inference-time steering by adding sparse, property-aligned vectors to LLM activations provides a robust, training-free, and composable mechanism for both single and multi-property molecular optimization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.