Reproduce “Probing Materials Knowledge in LLMs: From Latent Embeddings to Reliable Predictions” on OpenResearch

About this paper

Vineeth Venugopal, Soroush Mahjoubi, Elsa Olivetti

MIT researchers systematically evaluated Large Language Models in materials science, discovering that fine-tuning improves symbolic task performance by reducing output inconsistency, but numerical predictions are prone to 'confident hallucination' where models consistently output precise yet often inaccurate values. This work also revealed that for some numerical tasks, intermediate latent embeddings offer more accurate predictions than the final text output.

Key insights

• LLM behavior and reliability fundamentally differ by output modality: symbolic tasks show improved accuracy and reduced output entropy with fine-tuning, while numerical tasks often exhibit 'confident hallucination' (consistent but incorrect predictions) even after fine-tuning.
• For numerical prediction tasks like bandgap, internal latent embeddings in intermediate transformer layers can contain more accurate predictive information than the final text output, indicating a 'language modeling head bottleneck' during verbalization.
• Proprietary API-based models show substantial temporal performance instability, with up to 43% variation observed over 18 months, posing significant challenges for reproducibility in scientific research.

Summary by alphaXiv.