Reproduce “Releasing the Inner Jev: A Simple Recipe for Open Decision Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jonathan Geuter, Andrew Cheng, David Alvarez-Melis
The authors present a one-forward-pass method that converts questions and fixed answer options into probabilities by scoring answer-letter tokens, then uses supervised LoRA fine-tuning. On OpenDecisions, Gemma 4 31B reaches 79.6 Capability without fine-tuning, near Jev’s 80.1; after training it reaches 84.4, with Capability averaging Intelligence and Calibration. This is a development-set result, not an independent test.
Key insights
- Large instruction-tuned models can perform fixed-option decisions competitively even without fine-tuning: Gemma 4 31B scores 79.6 on OpenDecisions, close to Jev’s 80.1 after calibration.
- Fine-tuning the model’s representations matters more than training a new readout alone: LoRA with the initial decision head fixed reaches 84.9, while head-only training scores 78.9.
- The approach is tailored to evidence-based decisions, not general knowledge tests; Jev retains higher average accuracy on the six knowledge-heavy multiple-choice benchmarks reported.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.