Reproduce “Fine-Tuning Language Models to Know What They Know” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sangjun Park, Elliot Meyerson, Xin Qiu, Risto Miikkulainen
A bias-controlled framework rigorously measures and enhances Large Language Model metacognition, specifically self-knowledge, demonstrating that an Evolution Strategy for Metacognitive Alignment (ESMA) significantly improves discrimination between known and unknown information. This approach consistently boosted metacognitive sensitivity (d'type2) in open-source models, outperforming some proprietary LLMs and showing improvements from a sparse set of weight updates.
Key insights
- LLMs possess an intrinsic capacity for self-knowledge that can be rigorously measured and substantially improved through targeted fine-tuning.
- Evolution Strategies provide an effective optimization paradigm for complex, non-differentiable metacognitive objectives, addressing limitations of gradient-based methods.
- Metacognitive improvements are attributed to a sparse subset of weight changes, indicating a potentially localized yet spatially distributed mechanism within the model architecture.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.