Reproduce “Finetuning Language Models to Emit Linguistic Expressions of Uncertainty” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Arslan Chaudhry, Sridhar Thiagarajan, Dilan Gorur
The authors train Gemini 1.0 models to express uncertainty alongside question-answering responses, using confidence scores from the models’ own true-or-false judgments to assign verbal phrases. On held-out TriviaQA, Gemini 1.0 Small’s postfixed phrases achieve ECE 0.040 and Brier score 0.117, lower than prefixed or interleaved phrases. The evidence is limited to question answering and does not measure user outcomes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.