Reproduce “Mechanistic Interpretability of Code Correctness in LLMs via Sparse Autoencoders” on OpenResearch

About this paper

Kriz Tahimic, Charibeth Cheng

Researchers from De La Salle University applied sparse autoencoders to LLM representations, uncovering internal "directions" that predict code errors or enable corrections, thereby providing mechanistic insights into code correctness. The study found LLMs reliably detect incorrect code as anomalies and can be steered to correct errors with identified tradeoffs, demonstrating these mechanisms persist from base to instruction-tuned models.

Key insights

• LLMs exhibit an asymmetry in encoding code correctness, reliably detecting incorrect code as anomalies but lacking equally robust internal representations for positively affirming correct code.
• Successful code generation is critically dependent on the model attending to concrete test cases, rather than verbose problem descriptions, for determining code validity.
• Code correctness mechanisms learned during pre-training largely persist and remain effective after instruction-tuning, indicating refinement of existing knowledge rather than entirely new learning.

Summary by alphaXiv.